Toyota Research Institute: Transforming Robotics with Digital Innovation

My interest in the assistive robotics challenge at TRI grew directly from my other work with synthetic training data at CMU Robotics; it was another unique opportunity to apply digital precision to deeply human problems like independence and mobility. During my time at the Toyota Research Institute (TRI) from 2018 to 2021, I had the opportunity to support this mission by contributing to the sim-to-real (sim2real) robotics approach. While I wasn't a primary researcher, my role was to provide the high-fidelity digital foundation that fueled their machine learning models.

https://www.youtube.com/watch?v=LgaFkWCtSGU

Supporting the Sim2Real Pipeline

My primary contribution involved developing the accurate digital assets required for synthetic data generation. Because real-world data is expensive and slow to collect, TRI relies heavily on simulation to train robots. My work bridged the gap between physical objects and their digital twins:

• Millimeter-Level Accuracy: I transformed physical household objects into digital assets with extreme precision. This was vital because even a small discrepancy in a digital model can cause a robot to fail when it tries to grasp the real-world counterpart.

• Material Physics: I refined YAML and texture files to define complex material properties like reflectivity and flexibility. This ensured that the simulations didn't just look right, but behaved realistically within the training environment.

• Scaling through Simulation: By using the Godot game engine, I helped enable the simulation of millions of synthetic iterations. This provided the diverse training data necessary to improve the robots' computer vision and manipulation skills.

3D digital model of a domestic object for robotic simulation

Insights from the Research Environment

Being embedded in such a high-level research environment was an incredible learning experience. While I was there to provide technical art support, I gained deep insights into the practicalities of robotics:

• The Power of Synthetic Data: I saw firsthand how a robust sim2real pipeline can accelerate development, allowing researchers to test edge cases in a virtual environment that would be dangerous or impossible to replicate physically.

• Embracing Iteration: I learned that digital assets are never "finished" in a research context. I worked closely with engineers and computer scientists across three different teams, constantly adjusting models based on how the machine learning algorithms performed.

• Understanding Hardware Limits: Seeing how the "Punyo" soft bubble grippers interacted with the assets I created taught me a lot about the intersection of soft robotics and digital simulation.

Robotic hand with soft sensors gripping a household item

Reflections on Digital Innovation

My time at TRI fundamentally changed how I view the relationship between the digital and physical worlds. As a technical artist, I used to think of a "high-quality" model in terms of visual fidelity; however, I learned that for machine learning, quality is defined by physical truth.

• The Sim-to-Real Gap: I gained a deep appreciation for the "Reality Gap"—the subtle differences between a simulation and the real world that can cause a machine learning model to fail. I learned that my role wasn't just to make things look real, but to ensure the underlying data (like friction coefficients or light transport) was mathematically consistent enough for the AI to "trust" its training when it moved to physical hardware.

• Art as a Data Science: I realized that in the context of robotics, 3D modeling is essentially a form of data engineering. By adjusting a texture or a mesh, I was directly influencing the weights of a neural network. This taught me to look at my artistic workflow through the lens of a researcher, prioritizing how a computer "sees" an object over how a human does.

• The Feedback Loop: Perhaps the most significant thing I learned was the value of the iterative loop between the digital and the physical. Seeing the "Punyo" bubble grippers struggle with a specific object in the lab, and then going back to the digital twin to troubleshoot the physics, showed me that the most powerful innovations happen right at that intersection of bit and atom.