Engineers frequently utilize vision-language models to craft new designs for items like airplane or automobile parts. To determine how these parts will behave in real-life scenarios, they rely on computer-aided design (CAD) software to create 3D models for virtual testing. Researchers from MIT and other institutions have developed a system that can train a vision-language model to automatically transform 2D designs into CAD programs. This new method is more precise and requires less computation than previous techniques.
The improved AI-driven CAD generation could make rapid prototyping more efficient and cost-effective. Additionally, it could help engineers discover advantageous design options they might miss. The system creates new data by analyzing the model’s capabilities during the 2D to CAD program conversion process. It integrates both the model’s errors and successes into a dataset, helping the model learn to solve challenging problems.
“We aim for engineers to use our framework on a CAD model that’s not performing well, set a computational limit, and let the system enhance the model by learning from its own errors,” says lead author Giorgio Giannone, a research affiliate at MIT’s DeCoDE Lab and a principal research scientist at Red Hat. Co-authors include Anna Claire Doris, Amin Heyrani Nobari, Kai Xu, Akash Srivastava, and Faez Ahmed. The research was presented at the International Conference on Machine Learning.
Ahmed notes that while physical products start as CAD models, current models often produce overly simplistic shapes. The new method offers a way for image-to-CAD models to improve by learning from their errors, bringing reliable AI design tools closer to practical engineering.
The team is working to develop vision-language models (VLMs) that convert 2D images and text into Python code executable in CAD software. They identified the main challenge as a lack of diverse, high-quality CAD datasets for training. To address this, they employ data augmentation, generating data to enhance CAD generation models.
Instead of traditional methods, MIT researchers created the GIFT system (Geometric Inference Feedback Tuning) to produce data that improves a VLM’s performance on specific tasks. GIFT assesses a model’s strengths and weaknesses and generates data to tackle difficult CAD problems.
“We want data augmentation informed by the model itself,” Giannone explains. GIFT prompts the model to generate solutions multiple times, checking their accuracy to gauge problem-solving ability. Adjustments are made to near-correct solutions to teach the model, creating a new dataset that helps overcome typical difficulties.
GIFT’s approach uses “near-misses” and successes to generate model- and task-aware data augmentations. This expands the model’s understanding of CAD code generation without requiring human correction of its mistakes.
GIFT employs inference-time scaling to enhance a pre-trained VLM’s output without retraining. This method offers users control over computational resources, reducing costs. GIFT outperformed other methods, using only about 20% of the computation while improving accuracy.
Giannone notes they began with geometry, crucial for engineering tasks, but plan to expand GIFT to improve 3D model performance and manufacturability. Future goals include applying the system to larger models and diverse CAD generation tasks. The research received partial funding from the MIT-IBM Computing Research Lab.
Original Source: news.mit.edu
