Researchers and developers will be able to propel advancements across a variety of industries with the help of Hugging Face’s LeRobot open-source framework and NVIDIA AI and robotics technologies.
Hugging Face and NVIDIA established a partnership to unite their open-source robotics communities to expedite robotics research and development at the Conference for Robot Learning (CoRL) in Munich, Germany.
With the help of Hugging Face’s LeRobot open AI platform, NVIDIA Omniverse, and Isaac robotics technology, researchers and developers will be able to propel advancements in a variety of sectors, such as logistics, manufacturing, and healthcare.
Open-Source Robotics for the Era of Physical AI
The world’s industries are fast changing as a result of the advent of physical AI robots that can comprehend the physical characteristics of their surroundings.
Researchers and developers in robotics require open-source, extensible frameworks that cover the training, simulation, and inference stages of the development process in order to propel and maintain this rapid innovation. The most recent developments are easily accessible for use without requiring code redoing because to models, datasets, and workflows that are made available under shared frameworks.
More than 5 million machine learning researchers and developers use Hugging Face’s top open AI platform, which provides resources and tools to expedite AI development. With more than 1.5 million models, datasets, and applications freely available on the Hugging Face Hub, users may access and refine the most recent pretrained models and create AI pipelines using standard APIs.
Hugging Face’s LeRobot brings the Transformers and Diffusers libraries’ successful principles into the robotics space. In addition to designs for inexpensive manipulator kits, LeRobot provides a full array of tools for sharing data collecting, model training, and simulation settings.
NVIDIA’s AI technologies, simulation, and open-source robot learning modular architecture, like NVIDIA Isaac Lab, help speed up the LeRobot data collection, training, and verification workflow. To create a data flywheel for the robotics community, researchers and developers can share the models and datasets they have created with LeRobot and Isaac Lab.
Scaling Robot Development With Simulation
Physical AI is difficult to develop. Physics-based robotics depends on physical interaction data and vision sensors, which are more difficult to collect at scale than language models that employ vast amounts of internet text data. It takes a lot of time and effort to gather real-world robot data for dexterous manipulation across numerous tasks and settings.
This is made simpler by Isaac Lab, which is based on NVIDIA Isaac Sim and uses high-fidelity rendering and physics simulation to provide realistic synthetic environments and data, allowing robot training via demonstration or trial-and-error in simulation. A single demonstration can provide thousands of real-world experiences’ worth of training data thanks to Isaac Lab’s combination of parallel environment execution and GPU-accelerated physics simulations.
Imitation learning is then utilized to train a strategy using generated motion data. Following successful simulation training and validation, the policies are implemented on an actual robot and subjected to additional testing and fine-tuning to attain peak performance.
This iterative procedure ensures strong and dependable robotic systems by utilizing the scalability of simulated synthetic data and the precision of real-world data.
Developers and academics can build on each other’s work by sharing these datasets, policies, and models on Hugging Face, which speeds up advancements in the field.
“The robotics community flourishes when NVIDIA build together,” said Animesh Garg, an assistant professor at Georgia Tech. By using open-source frameworks like Hugging Face’s LeRobot and NVIDIA Isaac Lab, quicken the pace of research and development in AI-powered robots.
Fostering Collaboration and Community Engagement
The collaborative approach that is being suggested involves collecting data in Isaac Lab through teleoperation and simulation, then saving it in the LeRobotDataset standard format. A robot policy will be trained via imitation learning on data produced by GR00T-Mimic, and it will then be assessed using simulation. Finally, using NVIDIA Jetson for real-time inference, the verified policy is implemented on actual robots.
By demonstrating a physical picking setup with LeRobot software running on an NVIDIA Jetson Orin Nano, which offers a potent, small computing platform for deployment, the first stages in this collaboration have already been completed.
By fusing NVIDIA’s hardware, Isaac Lab simulation, and the Hugging Face open-source community, it could hasten advancements in AI for robotics,” said Remi Cadene, principal research scientist at LeRobot.
By supporting the most recent open models and libraries, including Hugging Face Transformers, optimizing inference for large language models (LLMs), small language models (SLMs), and multimodal vision-language models (VLMs), as well as VLM’s action-based variants of vision language action models (VLAs), diffusion policies, and speech models, all with strong, community-driven support, this work builds on NVIDIA’s community contributions in generative AI at the edge.
Hugging Face and NVIDIA are collaborating to speed up the work of the worldwide robotics research and development community, which is revolutionizing a variety of industries, including manufacturing, logistics, and transportation.
