Memories.ai Builds Standardized Egocentric Video Pipeline for Robot Training

Memories.ai is building a data pipeline for robotics companies that combines standardized video collection with AI-powered annotation to turn everyday egocentric footage into training data for robots. The company's models label every clip, with Memories-H0 handling hand pose and EgoAction identifying action segments.

Based in San Francisco, Memories.ai develops software and AI to help robots understand the physical world. It builds video-understanding models and data infrastructure, and it collects and annotates first-person video to supply high-quality training data to robotics firms. Egocentric video is captured from a wearer's perspective and is increasingly valued for robot learning because it shows hands, objects, and tasks from a human viewpoint.

The longer-term aim is to become a data-infrastructure partner for robotics and physical AI, not merely a data vendor. Memories.ai said its video data infrastructure platform is designed for physical AI, robotics, world models, vision-language-action models, and humanoids.

The platform offers a curated video data lake featuring egocentric video, along with custom annotation services and dataset tools for training robot learning models. Memories.ai obtains egocentric video through data collectors and partners, providing standardized capture hardware—a modular camera setup—and a uniform process for consistent footage collection.

Memories.ai then processes and annotates the data with its own models and makes it available through its infrastructure. The company has published research on its GitHub repository and at ICLR, NeurIPS, and EMNLP.

NVIDIA Cosmos helps Memories.ai realize its physical AI vision. Shawn Shen, co-founder and CEO, told The Robot Report that the company's vision is to "accelerate physical intelligence" and to be "part of the robotics game." He said Memories.ai aims to bring robotics into the physical world and will adopt whatever helps that vision arrive faster.

Shen said the company focuses on processing its own data rather than collecting video from a wide range of devices and sources. Once it receives data from partners, Memories.ai processes and annotates the footage itself.

The company said last month that it has used NVIDIA Cosmos to interpret video entering its data lake. It then curates that data to post-train Cosmos for customers' robotics and industrial applications, creating a loop that turns real-world video into models tailored to customer tasks.

Memories.ai said its pipeline has a modular design to support diverse tasks and camera views. Shen asserted that the company wants to grow with the industry as AI models mature for more robotics applications.

Shen said Memories.ai wants to be a "long-term partner" rather than just sell data. In the future, it wants to work with customers on models and infrastructure because customers are and will be limited by their resources, he said.

For robotics customers, Memories.ai plans to tailor data to their needs, offer exclusive or non-exclusive arrangements, and potentially expand its role into models and video infrastructure.