The false sense of continuity
Most data annotation work started in two dimensions. A
photo. A frame. A box drawn around something in a picture. That is a
well-understood problem with decades of tools and best practices built around
it.
When organizations start building physical AI, like robots
or self-driving vehicles, the instinct is to extend those 2D workflows into
three dimensions. Add a depth channel. Draw boxes in 3D instead of 2D. Give the
same team new tools.
This instinct is wrong, not because 3D annotation is
harder by degree, but because it is different in kind. The mental task, the
tooling, the quality checks, and the ways things go wrong are all fundamentally
distinct from 2D. Teams that treat 3D annotation as a difficulty upgrade
consistently produce training data that underperforms.
What 3D annotation actually requires
When someone labels an object in a 2D image, they are
answering a fairly contained question: where in this frame is this thing, and
what is it? The work stays in the coordinate space of the image.
When someone labels an object in a 3D point cloud, they
are answering something much bigger: where in physical space does this object
actually exist? What are its real dimensions? How is it oriented in
three-dimensional space? How is it moving over time?
This requires spatial reasoning that does not just come
from doing a lot of 2D labeling. The annotator has to understand that a lidar
point cloud is a sparse, partial snapshot of real objects, and reason about
what is actually there even when only part of the surface is captured.
It also means that errors carry physical weight. A
bounding box that is 10 pixels too wide in a 2D image is a minor quality issue.
A 3D bounding box that places a pedestrian 30 centimeters from where they
actually are is a safety-critical error in an autonomous vehicle dataset.
The sensor fusion layer
Physical AI systems almost never rely on just one sensor.
A self-driving car typically combines multiple cameras, a lidar system
generating millions of 3D measurements per second, radar sensors for velocity
and distance, GPS, and inertial units tracking motion.
These sensors do not all see the same thing at the same
time. They capture different aspects of the physical environment at different
speeds, from different positions, with different noise patterns and failure
modes.
This matters enormously for annotation. A label placed on
an object in the lidar data has to be consistent with how that same object
appears in the camera images captured at the same moment. A label that is
correct for one sensor's view can be wrong for another sensor's view of the
exact same scene, because each sensor sees reality differently.
Getting cross-sensor consistency right requires workflows
that 2D annotation platforms and teams simply were not designed for.
Why standard annotation tools fall apart
The annotation tool ecosystem was built for 2D computer
vision. Image segmentation, bounding boxes, keypoint labeling. These tools work
well for what they were designed to do.
When organizations try to use them for physical AI data,
they run into limits that are not bugs but fundamental design constraints.
These tools were not built to reason about physical space, time sequences
across multiple sensor streams, or the relationship between a sparse point
cloud and the actual objects it represents.
Teams find workarounds. They project 3D data into 2D
views, annotate there, and try to reconstruct 3D labels from 2D work. This
introduces errors at every step. The resulting 3D labels carry the limitations
of 2D spatial thinking applied to a 3D problem, and those limitations pile up
into training data that produces physically unreliable models.
The calibration problem nobody warns you about
Before 3D annotation can begin, there is a problem that
exists even upstream of annotation itself: sensor calibration.
Calibration is figuring out exactly where each sensor sits
relative to every other sensor, and how to translate coordinates from one
sensor's reference frame to another's. Without it, labels placed in one
sensor's coordinate frame will not line up with the correct location in another
sensor's frame.
This sounds like an engineering problem, not an annotation
problem. But poor calibration flows directly into annotation quality. An
annotator labeling a point cloud when the camera-lidar calibration is off will
produce annotations that look fine in the lidar view and are systematically
wrong in the camera view. That failure may not show up until a model trained on
the data is deployed and starts failing in specific geometric situations.
The quality of your 3D annotation is tied to the quality
of sensor calibration upstream. That is a dependency that 2D annotation
workflows do not have, and teams new to physical AI regularly underestimate it.
Temporal annotation: the dimension everyone forgets
Physical AI systems do not make decisions one frame at a
time. They act based on continuous sensor streams over time. A robot arm does
not execute a single movement; it executes a sequence of tiny adjustments, each
informed by what just happened. A self-driving car navigating an intersection
is processing a continuous stream of data over the whole approach.
Training data for physical AI has to capture this time
dimension. That means labeling not just what is in a given sensor frame, but
what action was in progress, what the intended outcome was, and where in a
multi-step sequence the frame falls.
This temporal layer is completely outside what 2D
annotation tools were designed to handle. It requires breaking continuous
action sequences into labeled phases. It requires making sure an object labeled
at one moment gets labeled with the same identity and physical dimensions at
every subsequent moment it appears. It requires labeling failures too, not just
successes, and marking exactly where in the time sequence the failure occurred
and why.
Building annotation capability that actually matches the
problem
The answer to the 3D annotation gap is not a better 2D
platform with a 3D feature bolted on. It is infrastructure, tooling, and team
capability built specifically for the physical world, for spatial data, time
sequences, multi-sensor fusion, and the level of precision that physical AI
systems require.
That means annotators trained in 3D spatial reasoning and
sensor physics, not just given a new set of instructions. It means quality
control built around the actual failure modes of 3D annotation: calibration
errors, temporal inconsistencies, cross-sensor label drift. It means annotation
guidelines that reflect the physical requirements of the models being trained.
Physical AI annotation is a discipline. Organizations that
treat it as a commodity service, interchangeable with 2D image labeling, will
build systems that reflect that misunderstanding. Organizations that invest in
building real capability will build systems that work.
The third dimension is not just an extra axis. It is the
axis where robots actually operate.
The false sense of continuity
Most data annotation work started in two dimensions. A photo. A frame. A box drawn around something in a picture. That is a well-understood problem with decades of tools and best practices built around it.
When organizations start building physical AI, like robots or self-driving vehicles, the instinct is to extend those 2D workflows into three dimensions. Add a depth channel. Draw boxes in 3D instead of 2D. Give the same team new tools.
This instinct is wrong, not because 3D annotation is harder by degree, but because it is different in kind. The mental task, the tooling, the quality checks, and the ways things go wrong are all fundamentally distinct from 2D. Teams that treat 3D annotation as a difficulty upgrade consistently produce training data that underperforms.
What 3D annotation actually requires
When someone labels an object in a 2D image, they are answering a fairly contained question: where in this frame is this thing, and what is it? The work stays in the coordinate space of the image.
When someone labels an object in a 3D point cloud, they are answering something much bigger: where in physical space does this object actually exist? What are its real dimensions? How is it oriented in three-dimensional space? How is it moving over time?
This requires spatial reasoning that does not just come from doing a lot of 2D labeling. The annotator has to understand that a lidar point cloud is a sparse, partial snapshot of real objects, and reason about what is actually there even when only part of the surface is captured.
It also means that errors carry physical weight. A bounding box that is 10 pixels too wide in a 2D image is a minor quality issue. A 3D bounding box that places a pedestrian 30 centimeters from where they actually are is a safety-critical error in an autonomous vehicle dataset.
The sensor fusion layer
Physical AI systems almost never rely on just one sensor. A self-driving car typically combines multiple cameras, a lidar system generating millions of 3D measurements per second, radar sensors for velocity and distance, GPS, and inertial units tracking motion.
These sensors do not all see the same thing at the same time. They capture different aspects of the physical environment at different speeds, from different positions, with different noise patterns and failure modes.
This matters enormously for annotation. A label placed on an object in the lidar data has to be consistent with how that same object appears in the camera images captured at the same moment. A label that is correct for one sensor's view can be wrong for another sensor's view of the exact same scene, because each sensor sees reality differently.
Getting cross-sensor consistency right requires workflows that 2D annotation platforms and teams simply were not designed for.
Why standard annotation tools fall apart
The annotation tool ecosystem was built for 2D computer vision. Image segmentation, bounding boxes, keypoint labeling. These tools work well for what they were designed to do.
When organizations try to use them for physical AI data, they run into limits that are not bugs but fundamental design constraints. These tools were not built to reason about physical space, time sequences across multiple sensor streams, or the relationship between a sparse point cloud and the actual objects it represents.
Teams find workarounds. They project 3D data into 2D views, annotate there, and try to reconstruct 3D labels from 2D work. This introduces errors at every step. The resulting 3D labels carry the limitations of 2D spatial thinking applied to a 3D problem, and those limitations pile up into training data that produces physically unreliable models.
The calibration problem nobody warns you about
Before 3D annotation can begin, there is a problem that exists even upstream of annotation itself: sensor calibration.
Calibration is figuring out exactly where each sensor sits relative to every other sensor, and how to translate coordinates from one sensor's reference frame to another's. Without it, labels placed in one sensor's coordinate frame will not line up with the correct location in another sensor's frame.
This sounds like an engineering problem, not an annotation problem. But poor calibration flows directly into annotation quality. An annotator labeling a point cloud when the camera-lidar calibration is off will produce annotations that look fine in the lidar view and are systematically wrong in the camera view. That failure may not show up until a model trained on the data is deployed and starts failing in specific geometric situations.
The quality of your 3D annotation is tied to the quality of sensor calibration upstream. That is a dependency that 2D annotation workflows do not have, and teams new to physical AI regularly underestimate it.
Temporal annotation: the dimension everyone forgets
Physical AI systems do not make decisions one frame at a time. They act based on continuous sensor streams over time. A robot arm does not execute a single movement; it executes a sequence of tiny adjustments, each informed by what just happened. A self-driving car navigating an intersection is processing a continuous stream of data over the whole approach.
Training data for physical AI has to capture this time dimension. That means labeling not just what is in a given sensor frame, but what action was in progress, what the intended outcome was, and where in a multi-step sequence the frame falls.
This temporal layer is completely outside what 2D annotation tools were designed to handle. It requires breaking continuous action sequences into labeled phases. It requires making sure an object labeled at one moment gets labeled with the same identity and physical dimensions at every subsequent moment it appears. It requires labeling failures too, not just successes, and marking exactly where in the time sequence the failure occurred and why.
Building annotation capability that actually matches the problem
The answer to the 3D annotation gap is not a better 2D platform with a 3D feature bolted on. It is infrastructure, tooling, and team capability built specifically for the physical world, for spatial data, time sequences, multi-sensor fusion, and the level of precision that physical AI systems require.
That means annotators trained in 3D spatial reasoning and sensor physics, not just given a new set of instructions. It means quality control built around the actual failure modes of 3D annotation: calibration errors, temporal inconsistencies, cross-sensor label drift. It means annotation guidelines that reflect the physical requirements of the models being trained.
Physical AI annotation is a discipline. Organizations that treat it as a commodity service, interchangeable with 2D image labeling, will build systems that reflect that misunderstanding. Organizations that invest in building real capability will build systems that work.
The third dimension is not just an extra axis. It is the axis where robots actually operate.