As a fan of tidying up, I gladly grabbed a dustpan and brushed up some spilled beans at the Toyota Research Lab in Cambridge, Massachusetts last year. The catch? I used a teleoperated pair of robotic arms with two-fingered pincers for hands.
Courtesy of Toyota Research Institute
Sitting before the table, using controllers like bike handles with extra buttons and levers, I could feel the sensation of grabbing solid items and sense their weight as I lifted them, but it took some getting used to.
After my tidying session, I went on with my lab tour. A few days later, Toyota sent me a video of the robot I’d operated sweeping up a similar mess on its own, using what it had learned from my demonstrations combined with several hours of practice sweeping inside a simulated world.
Autonomous sweeping behavior. Courtesy of Toyota Research Institute
Most robots, especially those handling tasks in warehouses or factories, can only follow preprogrammed routines. Getting them to adapt or improvise, like handling household chores, has proven to be a challenge. However, there are promising signs of change in the field, and the same algorithms that have boosted AI chatbots are helping robots learn more efficiently.
The sweeping robot I trained uses a machine-learning system called diffusion policy, developed in collaboration with researchers led by Shuran Song, to come up with the right action in a fraction of a second. Toyota is also exploring combining this approach with language models to enable robots to learn from videos, potentially transforming platforms like YouTube into powerful robot training resources.

