Ask an assistive robot “can you open this door?” and a naively prompted vision-language model answers from the camera image alone — with no access to the robot’s own tasks, skills and beliefs about the world, so it hallucinates. RACCOON grounds the answer in the robot’s internal state instead.
Inspired by Retrieval-Augmented Generation (RAG), RACCOON selects the relevant robot module — world model, tasks model, episodic memories — retrieves a state summary from it, and uses that to refine the prompt the LLM answers from.
In the paper’s evaluation this raises truthful answers from 15.9% to 82.5%. Selecting only the relevant context also beats passing every context source: 2.38 s to answer versus 4.54 s for the ablated variant.
Demonstrated on EDAN, DLR’s assistive robotic wheelchair system.