A factory worker can easily recall where she stored a partially assembled component the previous night and retrieve it quickly. However, robots working alongside her face challenges in developing similar “spatiotemporal” memory. MIT researchers have now created a long-term memory system that enables robots to quickly form and recall detailed mental models of complex, large environments.<\/p>\n
This advancement could eventually allow the worker to instruct a robot assistant simply to “fetch the component we started assembling last night.” The new method combines advanced mapping with comprehensive environmental descriptions that the robot collects over time. It allows the robot to access memory swiftly to respond to complex environmental queries in plain language.<\/p>\n
The memory framework delivers more accurate answers than current methods and operates quickly enough for real-time use by mobile robots. Beyond robotics, it could benefit augmented reality systems for maintenance workers in anomaly detection or help commuters with navigation.<\/p>\n
“For robots to work alongside humans effectively, they must communicate in the same language and reason about time and space like humans do,” says Luca Carlone, an MIT associate professor and principal investigator at the Laboratory for Information and Decision Systems. This method transforms a traditional map into a language-based one, making it easier for robots to understand and access using language.<\/p>\n
Carlone collaborated with lead author Nicolas Gorlo, an MIT graduate student, and Lukas Schmid, a former MIT research scientist now at the University of Technology Nuremberg. Their research was presented at the Conference on Computer Vision and Pattern Recognition.<\/p>\n
Spatiotemporal memory helps AI systems, like chatbots, answer complex questions and recall prior interactions. “Our goal is to create a spatiotemporal memory that enables AI-powered robots to remember real interactions and sensor observations,” Carlone explains, comparing it to ChatGPT but grounded in the real world.<\/p>\n
To build this memory framework, the MIT team combined computer vision and robotic mapping. While computer vision models describe objects in scenes, they often handle one annotation at a time. Robotic mapping can create 3D maps of environments but usually lacks detailed object descriptions or is computationally demanding.<\/p>\n
The researchers developed a method called Describe Anything, Anywhere, Anytime, at Any Moment (DAAAM), which merges these approaches. As a robot navigates, it notes detailed descriptions of objects, storing them in a spatially arranged 3D map. This allows the robot to recall that a red bicycle with a flat tire is at a specific location.<\/p>\n
Existing techniques that capture such detailed descriptions are too slow for real-time performance. DAAAM speeds up the process by aggregating nearby objects and using optimization to select key frames for annotation. This enhances computation speed significantly.<\/p>\n
The system attaches annotations to objects in specific map locations, allowing it to answer various queries about objects and locations. “We annotate each object once, enabling the framework to operate in large environments in real time,” Gorlo states.<\/p>\n
To retrieve information efficiently, the researchers employed a large language model (LLM) that uses various tools, reducing hallucinations and providing accurate responses in seconds. For example, if asked about a sculpture near an MIT building, DAAAM can use semantic search to retrieve relevant information.<\/p>\n
DAAAM demonstrated 21 to 53 percent greater accuracy than other methods, depending on the query type. In future work, researchers aim to enable the system to capture significant events and incorporate confidence levels into responses. “We aim to develop robots capable of assisting with any task,” Gorlo says, as they work on creating a generalist agent.<\/p>\n
This research received funding from the U.S. Army Research Laboratory and the Office of Naval Research. Carlone is currently on sabbatical as an Amazon Scholar, though the work described was conducted at MIT and is unrelated to Amazon.<\/p>\n