Generalist leverages human demonstrations to accelerate robot learning

Generalist is built on research into training robots with real-world data. The company's origins trace back to the paper "Universal Manipulation Interface: In-the-Wild Robot Teaching Without in-the-Wild Robots" (UMI), authored by teams from Toyota Research Institute (TRI), Columbia University, and Stanford University.

That paper introduced a data-collection platform using puppet-like end effectors operated by humans, with GoPro cameras capturing everyday tasks like dishwashing and object picking. The demonstrations become training data for robot foundation models, enabling collaborative robots to learn and take over these tasks much more quickly.

At the Automate trade show in June, Generalist demonstrated how its models could quickly generate policies for different cobot systems. The company showed Universal Robots (UR) arms folding and building cardboard boxes, while Flexiv arms were used to repair robot vacuums. According to a company social media post, the models impressed attendees not just by completing tasks, but by their ability to recover in real time when things went wrong.

Later, the company published a blog post titled "Towards Machines with a Thousand Hands" to highlight its work with various arms and grippers. In an interview, Samantha Castellanos, Generalist's founding mechanical engineer, explained the company's philosophy: "Our main goal is to make the best model in the world." She noted that the model is the key differentiator in a crowded space where competitors like Skild AI, Physical Intelligence, Field AI, and RLWRLD have raised over $4 billion combined.

Castellanos emphasized that all data collection and tool development is aimed at improving the model. "Everything we do is to make the model better." She also highlighted the company's approach to hardware: a one-degree-of-freedom gripper that is simple and robust. "Simple things are often the most robust," she said, adding that quick finger swaps minimize downtime.

Generalist moves from data collection to a working robot arm quickly. According to Castellanos, human data for the thousand-hands demonstration was collected in offices in Boston and California, ranging from 2 to 80 hours per task. Robot data collection was even shorter, often just a few minutes. For instance, training a tape-dispensing hand required only about 50 episodes and four minutes of robot data.

Castellanos admitted that the initial attempt with a screwdriver hand didn't work, but after training for more steps, it performed well. The key was proving that the model is gripper-agnostic and works with various end effectors.

When asked about industrial deployments, she said customers range from those wanting all-in-one solutions to those who already have UR arms and just need the model. The model improves weekly, and the company is still exploring the best business model. A significant differentiator is quick recalibration when things go wrong, as the model learns recovery behaviors from a large dataset.

Regarding future directions, Generalist has not yet run on a humanoid, but Castellanos believes it is feasible with the right integration. The company's models run on its own computer, which could be mounted on a mobile platform or humanoid. Notably, UMI co-authors Russ Tedrake and Ben Burchfiel have founded Walden Robotics, which applies large behavior models to mobile manipulators in industrial settings. Castellanos expressed optimism about the models' progress, saying they may soon need only a few minutes of data and can recover from edge cases.