Toyota Research Institute experiments with robots that hang from the ceiling and unfold to clean the kitchen

Researchers are using fleet learning and simulations to train robots to navigate one of the most complex environments: A home.

Visiting private homes in Japan inspired researchers to build a new domestic robot that moves around on the ceiling instead of the floor.

Image: Toyota Research Institute

Working in a factory is easy for robots with the structured environment and repetitive tasks that come with that job. Helping with housework is a much bigger challenge. Scientists at the Toyota Research Institute (TRI) are taking on that challenge by building new domestic robots and training them in a mock home.

Gill Pratt, the CEO of TRI and Kelly Kay, the Institute’s executive vice president and chief finance officer, gave a virtual tour of the TRI labs on Thursday. Max Bajracharya, the vice president of robotics and Steffi Paepcke, the senior user experience leader, explained the research and development process for building these robots.

The team is prioritizing user experience research, human-centered design, and ikigai–the idea that each person’s life should have deep meaning and purpose.

The institute’s philosophy is to build robots that take over tasks that have become too difficult for older adults instead of building a one-size-fits-all robot to take over all activities. One prototype is a gantry robot that unfolds from the ceiling to help with household tasks like a bat unfolding its wings.

The floor model looks like a praying mantis perched on a box. Researchers are using these models to develop capabilities.

“The robots that you see today are prototypes to accelerate our research, but they are not going to be turned into products any time soon,” Bajracharya said.

Field research for robotics experts

Paepcke said the team used the “genchi genbutsu” research technique which means, “go see for yourself,” to understand how to build domestic robots.

Before the pandemic started, researchers went to private homes in Japan to understand the daily challenges older adults and their caregivers face. Paepcke said that the goal was to understand which tasks people wanted help with as opposed to building a robot that does everything. Paepcke and her colleagues described the goal of their work to amplify human ability and help people continue to do tasks and activities that they find meaningful and enjoyable.

“A fully automated cooking robot might be physically helpful but emotionally detrimental,” she

Researchers used the home visits to move the cleaning robot from the floor to the ceiling.

Bajracharya said that the home visits showed that there was not much floor space available for a robot to move but that the ceiling provided more open real estate.

Researchers used virtual reality (VR) to teach the domestic robots how to clean a surface. A researcher performed the task in virtual reality to show the robot how to complete the task. Another challenge is helping robots understand how to distinguish between different surfaces such as wood, glass, and plastic.

Jeremy Ma, the Institute’s co-lead of the Robotics Fleet Learning Team, said the next challenge is to invent new ways for robots to learn quickly from one another. Ma’s team has built a mock home at the Institute’s Los Altos headquarters to do some of this training.

The research team is also working on fleet learning, the idea that when one robot is trained to do a task, this learning can be shared with other robots in the fleet.

“For us, the reality is for a robot to be successful, a robot has to be able adapt to multiple environments,” he said.

Learning to load the dishwasher

TRI also has a campus in Cambridge, MA, where the scientists work on manipulation. During the presentation, Russ Tedrake, PhD, showed off the lab that is full of robots standing over sinks full of dirty dishes. The goal is to train the robots to put cups, plates, and silverware in a dishwasher while also sorting out any trash. Tedrake said that his team has recently replaced pincer grips with bubble-end effectors. There are cameras behind the curved pads that watch how the pads change their shape to pick up different objects.

Tedrake said that data from the cameras powers deep learning algorithms to train a robot to understand what it is seeing and what to do next.

The robots learn by doing these tasks over and over again. In the video tour of the Cambridge lab, Tedrake said the team has recently started using computer simulations to train robots. The challenge is to give a robot enough data to be able to handle a corner case, a situation it has never seen before.

“We’re doing millions of experiments and simulations to make sure we’ve handled the edge case and the robot can work in a variety of situations,” said Pratt in a Q&A session with reporters on Thursday.

Woven City testing site in the works

In addition to this robotics work, Kay said that the Institute is working on three other areas of technology:

Driving: Building guardian technology and chauffeur technology
Accelerated materials design and discovery: Developing new materials for zero emission mobility
Machine-assigned cognition: Developing a better understanding of human behavior, particularly around decision-making

Two new groups within the TRI will support this work, including Toyota AI Ventures which invests in early-stage startups, and The Woven Group which will translate the Institute’s research into new products and services to be tested in the Woven City. The Woven City is a 175-acre site being planned at the site at the foot of Mt. Fuji. The TRI team meets every week with the team in Japan that is working on this project.

The Toyota Research Institute is training robots to sort dirty dishes and load the dishwasher with physical training and computer simulations
Image: Toyota Research Institute