Robots on city streets are becoming a more familiar sight, yet they are not yet the versatile helpers needed in kitchens or factories. A key hurdle is gathering data, as robots, like humans, learn effectively through experience. Teaching them various actions in different environments is a laborious and time-consuming process. “Using simulation as a training ground is a natural idea,” explains Russ Tedrake, a professor at MIT and a principal investigator at CSAIL. Although physics engines for robotics simulators have improved, creating rich and diverse content that mirrors real-world complexity remains a challenge.
AI agents, which are semi-autonomous programs capable of performing specific tasks, might provide the realistic virtual environments necessary for robot training. Researchers at MIT’s CSAIL and Toyota Research Institute have developed “SceneSmith,” a system that uses three agents to assemble 3D scenes. These recreations of spaces like restaurants and hotels are more detailed than previous systems, allowing robots to practice tasks before actual deployment, thereby saving engineers time on physical testing.
The agents rely on a vision-language model (VLM), specifically the advanced GPT-5.2, trained on extensive internet text and images. This model provides spatial knowledge, enabling a “designer” agent to create scene elements, a “critic” to assess realism, and an “orchestrator” to finalize designs. Once these agents conclude their collaboration, the scenes can be integrated into physics simulation software. “The system can construct 3D scenes like a human designer,” says Nicholas Pfaff, an MIT PhD student and lead author of a related paper.
SceneSmith allows users to request specific environments, such as a garage with particular items, creating rich virtual playgrounds where robots can learn skills. The system decorates scenes with significantly more items than previous methods, aiding in tasks like moving objects or placing items on surfaces. This reduces the need for trial and error in the real world, as robots can be evaluated in digital settings first.
To test realism, researchers inserted a pretrained robot policy into generated environments, checking if it could perform tasks like moving an apple to a cutting board. The experiments showed that SceneSmith’s environments endure physical interaction, not just visual inspection. Users also guided robots through these virtual spaces to test their functionality.
SceneSmith’s agents have specific roles in scene creation, from layout design to object placement, ensuring practical and high-quality environments. This approach allows for interactive elements like cabinets in scenes, which were less common in previous systems. SceneSmith outperforms other baselines, offering more realistic and object-rich scenes.
Over 200 users preferred SceneSmith, finding its visuals more realistic and accurate to prompts than other systems. The system is adept at generating diverse and detailed environments, though it requires significant time to produce each scene. With more computing resources, efficiency could improve. CSAIL engineers aim to expand the system’s capabilities, potentially incorporating deformable objects.
“SceneSmith represents a significant advance by providing an agentic framework for generating simulation-ready environments from text prompts,” says Jeremy Binagia, an applied scientist at Amazon Robotics. The system enhances the density and physical accuracy of objects in simulated environments, offering assets beyond fixed libraries.
Original Source: news.mit.edu
