The appealing promise of synthetic training

Simulation offers something the real world cannot: unlimited scale at nearly no additional cost. You can run a robot through a million training scenarios in a single day of compute time. You can generate edge cases that would be dangerous, expensive, or just impossible to collect in the real world. You can control every variable, keep things consistent, and produce training data at a pace no real-world collection program can match.

These advantages are real, and simulation genuinely belongs in the physical AI training toolkit. The mistake is not using simulation. The mistake is believing that simulation's advantages remove the need for real-world data. Teams that make this mistake discover, reliably and at significant cost, that the gap between simulated and real performance is not a minor tuning issue. It is a fundamental property of how simulation relates to physical reality.

 

What simulation actually is

A simulation is a computational model of the physical world. It represents the appearance of objects, how sensors behave, how rigid bodies move, how surfaces create friction, and how physics work during contact. It does all of this through mathematical approximations that are increasingly sophisticated but always remain approximations.

The gap between those approximations and actual reality is called the sim-to-real gap. It shows up differently depending on the physical system. For locomotion, teaching a robot to walk, balance, and navigate terrain, the gap is relatively manageable. The fundamental physics of bipedal or quadrupedal movement are well understood, simulation parameters can be tuned with relatively small amounts of real-world data, and reinforcement learning in simulation transfers reasonably well to physical hardware.

For manipulation, teaching a robot to grasp, handle, and precisely place objects, the gap is substantially harder. Contact physics are notoriously difficult to simulate accurately. The force dynamics at the moment a gripper closes on a physical object depend on the exact surface texture, give, and weight distribution of that specific object in that specific configuration, in ways that current physics engines approximate but do not fully capture.

 

Why locomotion and manipulation are different problems

To understand why simulation works reasonably well for locomotion but struggles for manipulation, you need to understand what each task actually requires.

Locomotion is primarily a dynamics problem. The robot needs to stay balanced, coordinate its limbs, and move in the intended direction. The physics involved, inertia, gravity, ground reaction forces, are well-modeled in modern physics engines. Domain randomization, the technique of varying simulation parameters randomly during training so the resulting policy is robust to uncertainty, works particularly well for locomotion because the real-world variance in relevant physics falls within the range a well-randomized simulation can cover.

Manipulation is a contact problem. The robot needs to interact with objects whose physical properties, weight, texture, give, friction, fragility, vary enormously and are often unknown until contact. The physics of contact between a robot gripper and a real object involve micro-scale surface interactions, deformation, and force distribution that are extraordinarily difficult to simulate accurately. A simulation might model a foam block and a hard plastic block as having different compliance values, but real foam has grain, compression history, and surface irregularities that no current simulation captures faithfully.

This is why physical AI systems trained mainly in simulation for locomotion transfer reasonably well, while systems trained mainly in simulation for complex manipulation routinely fail on contact-rich operations even when they perform perfectly in the simulator.

 

The calibration dependency

Simulation becomes more useful as it is more accurately calibrated to the real physical environment the system will operate in. Calibration means capturing the actual properties of real sensors, real objects, real environments, and real physics, and using that data to tune the simulation parameters.

A simulation of a factory floor calibrated against real sensor data captured in that factory will produce training data that transfers much more reliably to that factory than a generic simulation built from default physics parameters.

The practical implication is that real-world data is not an alternative to simulation. It is the input that makes simulation useful. The higher quality and the more representative your real-world sensor data, the more accurately your simulation can be calibrated, and the better simulation-trained policies will transfer to physical deployment.

This reframes the relationship: not synthetic versus real, but synthetic amplified by real. Every unit of high-quality real-world annotated data enables a larger volume of useful synthetic data.

 

The edge cases simulation generates and the ones it misses

Simulation has a genuine advantage for edge case generation. You can deliberately construct scenarios that are rare, dangerous, or logistically difficult to capture in the real world. Extreme weather conditions. System failure scenarios. Unusual object configurations. Near-miss situations that would be irresponsible to deliberately recreate with a physical robot.

This is valuable, but it comes with a catch: the edge cases simulation generates are the edge cases the simulation designers imagined. They are limited by what the designers could anticipate.

Real-world edge cases are different. They are the edge cases that actually happen. They emerge from the genuine unpredictability of physical environments, and they frequently include scenarios nobody anticipated. That is often exactly why they are edge cases.

A simulation-only training program covers the imagined edge cases well. A training program that includes real-world data from deployment environments covers the actual edge cases. The most robust physical AI systems train on both: simulation for the imagined edge cases at scale, real-world data for the actual edge cases that no one thought of.

 

What real-world data provides that simulation cannot

There are several properties of real-world physical data that simulation approximates poorly or not at all, and that nonetheless matter for physical AI systems operating in the real world.

Sensor noise and degradation: real sensors have noise patterns that vary with environmental conditions, usage patterns, and age. They develop biases, drift, and failure modes over time. Training on clean simulated sensor data produces models that have never learned to interpret the noisy, imperfect data that real sensors produce. The model encounters sensor degradation as an alien phenomenon rather than as something it was prepared for.

Material interaction: the physical properties of real materials, friction coefficients, surface give, deformation behavior, have natural variation that simulation approximates with fixed or randomized parameters. Real wood has grain. Real foam has compression history. Real cardboard has been weakened by handling. This natural variation is what the robot will encounter, and only real-world data faithfully represents it.

Environmental complexity: real environments have been shaped by human activity in ways that are difficult to model synthetically. Floors have subtle slopes and surface variations. Lighting has complex spectral properties and cast shadows. Objects have been worn, damaged, or modified by use. The authentic complexity of a real environment is a data source that simulation cannot fully replicate.

 

Building a training strategy that uses both

The practical conclusion is not that simulation should be avoided. It is that simulation should be understood accurately and combined with real-world data in a way that leverages the advantages of each.

Simulation works well for generating high volumes of training data for scenarios where the physics are well-modeled, locomotion, broad navigation, high-level path planning. It works well for generating dangerous or rare scenarios at scale. It works well for initial policy development before real-world data collection programs are up and running. But it should be calibrated against real data to be most useful.

Real-world data is essential for contact-rich manipulation tasks, for sensor noise and degradation coverage, for deployment-environment-specific calibration, and for the actual edge cases that emerge from real operational environments. It is the ground truth that simulation approximates.

Physical AI programs that treat these as competing approaches, picking one or the other, consistently underperform compared to programs that build both and use each where it fits best.

Simulation teaches a robot to walk. Real data teaches it to work. Physical AI demands both.