1. Why Embodied AI Has the Hardest Annotation Problem in ML
Text preference annotation asks: between these two responses, which is better? The annotator reads both, applies judgment, selects one. The output is a token sequence. The stakes of a wrong annotation are a slightly worse model response. The annotation can be redone. Nothing breaks.
Robotics annotation asks: between these two motor control trajectories, which is safer, more efficient, and more likely to generalize to novel object poses? The annotator watches a 30-second video of a robotic arm attempting a grasp, or a bipedal platform navigating an obstacle course, and must evaluate not just whether the task succeeded but whether the underlying trajectory is physically sound. The output is a continuous action sequence in 3D space. A wrong annotation trains the policy toward motions that are visually smooth but dynamically infeasible, or that succeed in the specific test configuration but catastrophically fail on any variation.
The stakes are categorically different. Boston Dynamics Atlas operating at dynamic balance limits, Figure AI's 02 performing dexterous manipulation in a factory environment, Physical Intelligence's π0 policy executing household tasks, Tesla Optimus in an assembly line — "good enough" annotation on these systems means a broken end-effector, a joint limit violation that damages the actuator, or a workspace collision that injures a co-worker. The annotation quality bar for embodied AI is not the same as the bar for text preference data, and the annotator profile that meets that bar is completely different.
There are three structural reasons embodied AI annotation is harder than any other ML annotation domain. First, the output space is continuous — a trajectory is not a discrete choice but a temporal sequence of joint angles, velocities, and forces, and quality lives in the dynamics between frames, not in any single frame. Second, safety evaluation requires physical intuition that cannot be developed by watching videos — a joint limit violation looks like a smooth motion to someone who has never seen an actuator fail. Third, the sim-to-real gap means that trajectory quality in simulation is not evidence of trajectory quality on hardware, and the annotator who cannot reason about transfer is not evaluating the thing that matters.
2. The Five Annotation Dimensions Unique to Robotics and Embodied AI
Robotics annotation decomposes into five dimensions that require distinct evaluation criteria and distinct annotator expertise. Collapsing them into a single preference rating produces signal that is neither actionable nor interpretable.
(a) Trajectory quality
Was the motion path efficient, collision-free, and within joint limits? Trajectory quality is not visual smoothness — it is dynamic feasibility. A motion that looks smooth in a rendering can violate joint velocity limits, exceed torque capacity, or accumulate contact forces that will damage the end-effector over repeated cycles. Evaluating trajectory quality requires a robotics background: understanding of Jacobian singularities, knowledge of the specific platform's joint limits and safety margins, and the ability to recognize when a visually acceptable motion is mechanically problematic. Mechanical engineers and robotics engineers are the appropriate annotator profile. Generalists rating "does it look smooth" are annotating visual rendering quality, not trajectory quality. Rated 1–5, with required rationale for any rating below 3.
(b) Task completion accuracy
Did the robot accomplish the stated goal? This dimension uses a binary plus partial credit rubric: 0 (complete failure), 0.5 (partial completion — goal partially achieved or achieved with significant degradation), 1 (full completion within acceptable tolerance). Partial credit is not optional — a grasp that picks up the object but immediately drops it is not the same as a grasp that never contacts the object. The partial credit score captures meaningful information about where in the task the policy is failing and what kind of additional training data would address it. Task completion annotation is less technically demanding than trajectory quality — it requires domain context but not necessarily deep robotics engineering expertise.
(c) Safety flagging
Any motion that risks hardware damage, workspace collision, or human proximity violation must be flagged. Safety flagging is the highest-stakes annotation dimension and has the strictest IAA requirement: κ ≥ 0.85, with zero tolerance for missed flags. A trajectory that approaches a joint limit without triggering a safety flag is a false negative with direct cost implications — actuator replacement costs range from thousands to tens of thousands of dollars, and the operational downtime compounds that. A trajectory that operates near human co-workers without triggering a human proximity review is a false negative with safety-critical implications. Safety flags are binary (Y/N) with a required note specifying the risk: "joint angle at 94% of soft limit on axis 3 — high repeated-cycle wear risk" is a usable flag. "Looks a bit close" is not. Disagreements on safety flags are never resolved by majority vote — see Section 6.
(d) Generalization likelihood
Given this trajectory on this specific object configuration, would it generalize to object pose variation of ±15°? This dimension captures the sim-to-real problem at the annotation level. A trajectory that succeeds on a specific object at a specific pose is not evidence of a generalizable policy — it may be a perfectly fitted solution to that exact configuration that fails on any variation. Generalization likelihood requires simulation intuition: the annotator must mentally simulate what happens to the grasp geometry as the object rotates, what happens to the contact forces as friction changes, whether the approach trajectory has enough clearance for pose variation within the expected tolerance. This is the dimension with the highest expertise requirement and the lowest crowdworker IAA (κ < 0.25 without robotics background). Rated 1–5.
(e) Human-robot interaction quality
For HRI tasks — assistive robotics, collaborative manufacturing, service robotics — this dimension evaluates proximity comfort (did the robot approach at a speed and distance that a human co-worker would find comfortable and predictable), communication clarity (were the robot's intentions legible through motion or signaling), and handoff safety (was the object transfer executed in a way that minimized collision and force transfer risk). HRI quality annotation requires annotators with human factors engineering or occupational therapy backgrounds for assistive contexts, and safety engineering credentials for industrial co-robot environments. A mechanical engineer who has never observed human-robot collaboration is not the right annotator for this dimension.
3. Why Crowdworkers Cannot Annotate Robotics Data
Crowdworker IAA on robotics annotation tasks is κ < 0.30 for trajectory quality and κ < 0.25 for generalization likelihood. For context, κ = 0.30 is below the threshold considered "fair agreement" in annotation literature — it is barely above chance. These numbers are not a calibration failure or a rubric design problem. They are a structural consequence of three specific gaps that better instructions cannot close.
(a) No physics intuition
Crowdworkers evaluate visual smoothness, not dynamic feasibility. A trajectory that looks fluid in a rendered video may be driving the end-effector through a configuration that accumulates contact stress, approaches a kinematic singularity, or requires torques that the actuator cannot sustain at thermal limits. Crowdworkers cannot see any of this — they see a smooth motion and rate it positively. The gap between visual smoothness and dynamic feasibility is the gap between κ = 0.28 and κ = 0.72. It cannot be closed by showing crowdworkers more examples or writing more detailed rubrics. Physics intuition is acquired through robotics coursework and hands-on hardware experience, not through annotation training.
(b) No domain safety vocabulary
Crowdworkers do not know what a joint limit violation looks like. They do not recognize a workspace collision risk from a trajectory that passes within 50mm of a fixed obstacle — the safety margin for a specific platform at a specific speed profile is not visible from a video. They cannot identify a human proximity violation in a co-robot context because they have no reference for what proximity constraints apply to the specific industrial standard (ISO 10218-2, TS 15066). A crowdworker watching an Atlas demo cannot tell you whether a specific stance configuration is within the platform's stability margin — and neither the annotation rubric nor additional training will give them that knowledge.
(c) No sim-to-real intuition
Crowdworkers cannot assess whether a trajectory that succeeds in IsaacGym or MuJoCo will transfer to real hardware. The sim-to-real gap is not visible in simulation footage — it manifests as contact dynamics that differ from the physics model, friction that varies across object materials, actuator compliance that does not match the rigid-body assumption. An annotator without hands-on hardware experience has no basis for estimating whether a simulated trajectory's generalization likelihood score means anything about real-world performance. This is why generalization likelihood reaches κ < 0.25 with crowdworkers — they are rating something they have no mechanism to evaluate.
For the general case against crowdsourcing for domain-specific annotation, including a framework for evaluating annotator expertise requirements, see domain expert annotators vs. crowdsourcing for AI training.
4. Domain-Specific Considerations by Robotics Application
Embodied AI is not a single domain. Manipulation, locomotion, HRI, and sim-to-real evaluation have different annotation requirements and different annotator credential profiles. Running them through the same annotator pool produces a dataset that is adequate for none of them.
Manipulation (grasping, assembly, dexterous tasks)
Manipulation annotation requires mechanical intuition — contact forces, friction coefficients, grasp geometry, and the relationship between approach trajectory and contact stability. A robotics engineer or mechatronics graduate with hands-on manipulation experience is the minimum qualification. The Physical Intelligence π0 policy, trained with RLHF-style human feedback on dexterous manipulation tasks, represents the current frontier of what well-annotated manipulation training data can produce — and it required annotators who could evaluate whether a specific finger placement would produce stable contact across object pose variation, not annotators who could evaluate whether the hand "looked natural."
Navigation and locomotion (bipedal, wheeled, aerial)
Locomotion annotation focuses on gait stability, obstacle clearance margins, and recovery behavior quality. For bipedal platforms (Boston Dynamics Atlas, Agility Robotics Digit, 1X Technologies NEO), stability annotation requires understanding of zero-moment point dynamics, foot placement under perturbation, and what recovery behaviors look like versus what precursor-to-fall behaviors look like. IAA targets for stability annotation should be κ ≥ 0.70 — lower than safety flagging (κ ≥ 0.85) but still requiring annotators who can distinguish between a stable recovery and a trajectory that happens not to fall in this specific trial. For wheeled platforms, obstacle clearance margin annotation requires 3D spatial reasoning about trajectory-to-obstacle distance that is not perceptually obvious from standard camera viewpoints.
Human-robot interaction (assistive, collaborative, service)
Assistive robotics annotation — exoskeletons, rehabilitation platforms, elder-care robots — requires licensed occupational therapists or physical therapists who can evaluate whether a motion pattern is appropriate for the specific patient population. The annotation question is not "does the robot complete the task" but "does this interaction pattern support the human user's dignity, safety, and therapeutic goals." For industrial co-robot environments (Apptronik Apollo in warehouse settings, Tesla Optimus in manufacturing), safety engineers with ISO 10218 and TS 15066 familiarity are the appropriate annotator profile. Human factors engineers are the right profile for service robotics contexts where interaction naturalness and predictability drive adoption.
Sim-to-real transfer evaluation
Generalization likelihood annotation is meaningless from annotators who have only worked in simulation. The sim-to-real gap is experienced, not calculated — you develop intuition for which simulated behaviors will transfer by watching policies fail on hardware in specific ways: contact dynamics that don't match the rigid-body model, friction that varies between the simulation material assignment and the actual object surface, actuator compliance that absorbs impact differently than the simulated rigid joint. Annotators evaluating sim-to-real transfer must have hands-on hardware experience on platforms similar to the deployment target, not just simulation expertise in IsaacGym or MuJoCo.
Building a robotics AI system and need expert trajectory annotation?
Our Starter Pack includes robotics engineers and mechatronics specialists matched to your platform — 25–50 expert-annotated trajectories to calibrate your pipeline before full-scale collection.
Check out our Starter Pack →5. Practical Annotation Schema for Robotics Tasks
A production annotation schema for robotics trajectories collects five fields per annotation unit. The time investment is higher than any other annotation type: a 30-second trajectory clip can take 10–15 minutes to evaluate properly. Budget for this — if your annotators are completing robotics annotation faster than 10 minutes per 30-second clip, they are not evaluating all five dimensions.
Task specification and environment setup
The stated task goal ("pick up the red cylinder and place it in the bin"), the object set present in the scene (including materials and approximate masses if available), workspace constraints (clearance zones, human proximity areas, fragile surfaces), and the starting configuration (robot pose, object placement). This field provides the context the annotator needs to evaluate task completion and safety — without a clear statement of what the robot was supposed to do and what the environment constraints were, trajectory evaluation is guesswork.
Trajectory recording
The action sequence (joint angle commands or end-effector poses at each timestep), timestamps, and sensor readings if available (force-torque sensor data, contact sensor readings, IMU for locomotion tasks). The trajectory recording is what the annotator actually evaluates — video alone is insufficient for trajectory quality and generalization likelihood annotation. Annotators evaluating trajectory quality should have access to joint angle plots alongside video to identify limit approaches that may not be visible in the rendered view. Force-torque data provides direct evidence for contact force annotation that video alone cannot supply.
Per-dimension ratings
Trajectory quality (1–5, required rationale for ≤3), task completion (0 / 0.5 / 1, with description of what succeeded and what failed for partial credit), safety flag (Y/N — if Y, required note specifying the risk category: joint limit approach, workspace collision risk, human proximity violation, or other hardware damage risk), generalization likelihood (1–5, required rationale explaining which aspects of the trajectory would or would not transfer to pose variation). All five dimensions should be rated independently — collect them as separate fields, not as an aggregate score.
Annotator confidence
A self-reported confidence level (high / medium / low) for each dimension rated. Low-confidence ratings should trigger adjudication before the annotation is used in training — a safety flag from a low-confidence annotator is not the same as a safety flag from a high-confidence annotator with senior robotics engineering experience. Confidence fields provide the adjudication trigger that keeps uncertain annotations from entering the training pipeline as clean signal.
Rationale for safety flags
Any safety flag must include a structured rationale: the specific timestamp(s) where the risk manifests, the risk category (joint limit / workspace collision / human proximity / hardware wear), the specific mechanical or safety concern, and the annotator's recommendation (retry with modified approach trajectory / flag for senior review / do not include in training). Unstructured safety flag notes ("looks unsafe") are not sufficient — the rationale must be specific enough that a senior robotics engineer can evaluate it without watching the video.
Note: robotics annotation is 5–10× more time-intensive than text preference annotation. A 30-second manipulation trajectory requires 10–15 minutes of expert evaluation to assess all five dimensions properly. A locomotion sequence with perturbation recovery events may take longer. Budget and timeline planning for robotics annotation should start from this baseline, not from text annotation cost benchmarks. For a general treatment of annotation cost estimation, see scaling RLHF to 10,000+ annotations.
6. Calibration and Inter-Annotator Agreement for Robotics Tasks
Calibration for robotics annotation requires a pilot set of 20–30 trajectories specifically designed to cover the failure modes most common on your robot platform. A generic calibration set is not adequate — a manipulation platform failure mode set (grasp approach angle failures, contact force overruns, object slip on placement) is different from a locomotion platform failure mode set (gait instability under perturbation, obstacle clearance failures, recovery behavior degradation). Build the calibration set from logged failures on your hardware, not from synthetic examples or simulation data.
IAA targets by dimension:
- →Task completion: κ ≥ 0.65 — the partial credit rubric introduces legitimate disagreement on marginal cases; κ = 0.65 is achievable with trained robotics annotators and indicates the rubric is well-defined
- →Trajectory quality: κ ≥ 0.70 — higher bar than task completion because quality annotation requires more specific technical agreement on what constitutes a limit approach or dynamic infeasibility
- →Safety flags: κ ≥ 0.85 — the highest bar in the schema. False negatives on safety flags are a hardware and safety liability. An annotator pool at κ = 0.70 on safety flags is producing inconsistent safety signal; scaling annotation volume amplifies this inconsistency, it does not average it out
⚠ Crowdworker pipeline
- κ < 0.30 trajectory quality
- κ < 0.25 generalization likelihood
- No physics intuition — visual smoothness only
- No safety vocabulary — joint limits invisible
- No sim-to-real intuition — transfer not assessable
✓ Domain expert panel
- κ ≥ 0.70 trajectory quality
- κ ≥ 0.65 task completion
- κ ≥ 0.85 safety flags
- Physics intuition from hardware experience
- Sim-to-real transfer assessable
Adjudication protocol for safety flags: Safety flag disagreements require physical review by a senior robotics engineer — never average or use majority vote on a safety call. If two annotators flag a trajectory and one does not, the trajectory is held for senior engineer review before it enters the training pipeline in any direction (as a positive or negative example). The cost of a false negative safety flag — a trajectory that trains the policy toward hardware-damaging behavior — is not recoverable from downstream model evaluation. It must be caught at the annotation layer.
For annotation prompt and rubric design that produces higher IAA from the start, see how to write better prompts for RLHF annotators. For evaluation methodology connecting annotation quality to downstream reward model quality, see how to evaluate RLHF reward models. For overlap with multimodal and perception annotation in robotics vision stacks, see multimodal RLHF for vision-language models.
7. Getting Started: Minimum Viable Dataset for Embodied AI
Dataset sizing: 300–500 trajectory annotations per manipulation primitive. Do not mix grasp, place, and assembly annotations into a single dataset — they are different distributions with different failure modes and different generalization challenges. A grasp trajectory annotation pool and a place trajectory annotation pool should be collected separately, annotated separately, and evaluated separately. Mixing them produces a dataset that accurately represents neither primitive and provides weaker training signal than a smaller, purer collection.
Annotator coverage: 3 annotators per trajectory minimum for safety flags, 2 for quality. Single-annotator safety flag coverage is not acceptable — a hardware or human safety call requires at least two independent expert reviews. For quality dimensions (trajectory quality, generalization likelihood), two annotators with adjudication on disagreements is the minimum viable coverage. Three annotators across all dimensions is the recommended default for manipulation tasks where annotation disagreement is common on grasp geometry evaluation.
Cold start: 20-trajectory calibration pilot before full collection. Run a calibration pilot on 20 trajectories drawn from your platform's known failure mode distribution before committing to a full annotation run. The pilot tells you three things: whether your annotator pool reaches IAA targets on your specific task, whether your annotation schema produces the fields your training pipeline actually needs, and whether your video rendering and data packaging makes the relevant trajectory information legible to annotators. Starting a 500-trajectory annotation run without a calibration pilot is the most expensive way to discover that your joint angle plots are in the wrong coordinate frame.
Physical Intelligence's π0 policy — trained with RLHF-style human feedback on dexterous manipulation — is the clearest current evidence that expert trajectory annotation at scale produces meaningfully better policies than simulation-only training. The quality of the feedback signal is what separates a policy that generalizes to novel object geometries from one that overfits to the training configuration. The annotation infrastructure is where that quality is built.
Decision rule: If the robot will be in physical contact with humans, operating around expensive equipment, or executing tasks where a trajectory failure could damage hardware, expert annotation is a safety requirement, not a preference. A 67% real-world success rate on a grasp task is not a model quality problem — it is an annotation quality problem. The sim-to-real gap that produces it is invisible to automated metrics and invisible to crowdworkers. It is only visible to robotics engineers with hands-on hardware experience evaluating trajectories against the five dimensions above.