- A robot video proves almost nothing on its own. Autonomy, take count, object novelty and cycle time are the four facts that decide whether a clip is evidence.
- The data problem is the whole field. Open X-Embodiment pooled 60 datasets from 34 labs to reach roughly a million trajectories across 22 robot bodies, which is a rounding error next to a text corpus.
- Teleoperation is not cheating, it is the data pipeline. Companies say so openly, and the interesting question is which parts have graduated to autonomy.
- Robot evaluation runs on tiny samples. Published policy comparisons work with 10 or 50 trials, which is why two papers can disagree without either being wrong.
- Cross-embodiment transfer is real but modest. RT-X improved several robots by borrowing experience from others, which is a long way from one model driving any body.
- The safety standard moved in February 2025 after nearly eight years of work, adding robot classes and cybersecurity to a document that had stood since 2011.
The clip is twelve seconds long. A humanoid picks up a mug, turns, and sets it in a dishwasher rack without dropping it. The comments split immediately between people saying this changes everything and people saying it is remote controlled. Both groups are guessing, because the video does not contain the information that would settle it.
That missing information is the subject here. Robotics has a measurement culture that is nothing like the one around language models, for reasons that are physical rather than cultural, and the result is a field where the published numbers are honest and the public perception is unmoored from them. What follows is how to read the gap.
Why is there no internet for robots?
Because text was already written down and robot behaviour was not. That sentence sounds glib and it is the entire structural problem.
A language model trained on the public web inherited trillions of tokens that humans produced for their own reasons, at no cost to the lab. There is no equivalent archive of a hand closing around a mug at the right pressure. Every trajectory has to be produced deliberately, by a person driving a robot or by a robot trying and failing, on hardware that wears out.
The scale of the response tells you the scale of the problem. Open X-Embodiment was assembled by pooling 60 existing robot datasets from 34 laboratories, coordinated across 21 institutions, covering 22 robot bodies and 527 skills across 160,266 tasks, standardised into roughly a million trajectories. That was a landmark act of cooperation in a field that does not cooperate easily. It is also, in data terms, tiny.
Three strategies exist for filling the hole, and every serious lab is betting on some mixture. Collect teleoperation logs at industrial scale. Generate experience in simulation and hope it transfers. Pretrain on human video and hope the physics comes along for the ride. None of them has produced the moment where the curve bends, which is why the debate is still live.
What actually transfers from simulation to hardware?
Geometry and gross motion transfer well. Contact does not.
A simulator is very good at where things are and how a limb swings. It is much worse at the moment two surfaces touch, because friction, slip, compliance and material variation are approximated by parameters somebody chose. Rigid objects with clean geometry survive the trip. Cloth, cables, liquids and anything requiring a controlled squeeze arrive in worse shape than they left.
The table below sorts task classes by how well they hold up, with the honest caveat attached to each. The point is not to publish a leaderboard, it is to show that the phrase robot foundation models covers wildly different levels of maturity.
| Task class | How results are usually produced | What holds up on hardware | The caveat that gets dropped |
|---|---|---|---|
| Locomotion over terrain | Simulation at massive scale, then transfer | Strong. Walking and recovery generalise well | Walking is not manipulating. A robot that hikes cannot fold a towel |
| Pick and place, rigid objects | Teleoperation data plus imitation learning | Good on trained object sets, weaker on novel shapes | Success rate is usually reported on the objects the policy saw |
| Deformable manipulation | Real hardware demonstrations, limited simulation help | The weakest area. Cloth and cable defeat contact models | Demos show the successful attempt, not the attempt distribution |
| Long horizon multi step tasks | Hierarchical policies, high level planner plus low level skills | Improving, brittle to any step failing | Errors compound, so a 90 percent step is a 35 percent ten step task |
| Human interaction and handover | Scripted or teleoperated in most public demos | Hard to evaluate, easy to stage | Safety constraints mean the interesting version is rarely shown |
That fourth row is worth pausing on because it is arithmetic rather than opinion. If each step in a ten step task succeeds independently 90 percent of the time, the whole task succeeds 0.9 to the tenth power, which is about 35 percent. Long horizon work is where robotics is genuinely hard, and it is also where the marketing lives.
How do you read a robot demo honestly?
With five questions, in this order. None of them requires expertise, and most videos fail on the first two.
Is it autonomous?
Teleoperation is common, disclosed less often than it should be, and not shameful. It is how the training data gets made. Sanctuary described its approach to IEEE Spectrum in exactly those terms: begin with high quality teleoperation, where a pilot wears a rig that transmits their movements to the robot, then build increasing layers of autonomy on top until the system reaches full independence. Frequently repeated actions such as opening a door or grasping a handle are automated first, then chained.
So the question is never whether teleoperation is involved somewhere. It is which layer you are being shown. A useful follow up: which specific subtasks in this clip run without a pilot?
Is it one take?
A single successful attempt tells you the task is possible, not that it is reliable. Publication practice in the field is to report a success rate over trials. Video practice is to show the good one. When a company reports a percentage, ask over how many attempts, and whether failures were reset by a human.
Is the object novel?
Most reported success rates are measured on objects in the training distribution. A policy that stacks the same six mugs at 85 percent may collapse on a mug with a different handle. The interesting demo is the one where somebody hands the robot an object it has never seen, on camera, chosen by the person handing it over.
What is the cycle time?
Compare it to a human doing the same thing. A robot that folds a shirt in four minutes has demonstrated a research result, not a laundry business. Speed is where a great many impressive capabilities quietly fail their economic test, and it is the number most often edited out of a clip.
How many trials produced the number?
Fewer than you think, and this is the single most underappreciated fact about robotics research. A paper on comparing imitation learning policies states the constraint outright: policy comparison is limited to a small feasible sample size, on the order of 10 or 50 trials, because each trial costs human effort and the policies are slow to run. The same paper warns that extending a test with extra trials after seeing the results risks p-hacking, and proposes a sequential test that cuts the number of trials needed by up to 32 percent.
Sit with that. When a robotics result says a policy improved from 60 percent to 75 percent, the underlying data may be 15 successes out of 20 against 12 out of 20. That is not fraud. It is what physical evaluation costs. It does mean a single result should move your beliefs a lot less than a benchmark score in software would.
Can anyone trust cross-lab comparisons?
Not yet, and the field knows it. The evaluation problem in robotics is worse than the reproducibility problem in machine learning, because the environment is not shipped with the code. Lighting, table height, the specific gripper, the exact mug: all of it is part of the experiment and none of it is in the repository.
RoboArena is the most interesting response so far. Rather than a centralised benchmark with fixed tasks, it distributes evaluation across 7 academic institutions on a shared robot platform, with double blind pairwise comparisons, and reports over 600 pairwise real-robot episodes across 7 generalist policies. The argument is that a crowd sourced ranking is more accurate than a centralised one precisely because no single lab's environment dominates.
The deeper lesson generalises past robotics. A score is a property of a model inside a harness, not of the model, which is the same conclusion we reached looking at how agent benchmark scores move with the compute budget they were measured at. In robotics the harness includes a room.
What did the foundation model wave actually deliver?
Positive transfer, at a scale worth taking seriously and smaller than the phrase implies.
The RT-X result from Open X-Embodiment is the clean statement: a high capacity model trained on the pooled dataset improved the capabilities of multiple robots by drawing on experience from other platforms. Experience gathered on one arm helped a different arm. That had not been reliably shown before, and it is the reason the cross-embodiment framing took over.
On the dexterity side, the pi-0 model pre-trained on more than 10,000 hours of robot data and fine-tuned to tasks including laundry folding, clearing a table, loading a microwave, stacking eggs into a carton and bagging groceries. The laundry policy runs a genuinely long chain: fetch from the dryer, pack a hamper, move it to a table, fold each item. The paper reports that pre-training helps most on tasks resembling the pre-training data, with the pre-trained model sometimes twice as good as the version without it.
Read that carefully and you get the honest summary of where the field is. Pretraining helps most where the new task looks like the old data. That is a useful property and it is not generalisation in the sense the word implies when a language model answers a question nobody wrote down. The same distinction between predicting a world and acting in one runs through how world models predict reality, and robotics is where the difference stops being philosophical.
None of this is an argument that robotics is stalled. Locomotion in the last five years went from remarkable to routine. The argument is narrower: manipulation results are reported under conditions that a video cannot convey, so a video is the wrong instrument for forming a view.
Why do two robotics papers disagree so often?
Because they are answering questions that only look identical. Three variables move underneath almost every disagreement, and once you can name them the literature stops seeming contradictory.
The object set. One paper evaluates on objects sampled from its training distribution, another on held out shapes. Both call the metric success rate. The numbers are not comparable and neither team did anything wrong.
The reset policy. When an attempt fails, does a human reposition the object before the next trial? A generous reset turns a fragile policy into a competent looking one, and the detail lives in an appendix if it lives anywhere.
The stopping rule. This is the subtle one the policy comparison work names directly. If a team runs 20 trials, sees a promising split, then runs 20 more, the resulting p value no longer means what it says. The proposed sequential test exists precisely so researchers can extend a comparison without invalidating it, and its existence tells you the practice it is correcting was common.
Stack those three and a gap between two published numbers can be entirely procedural. This is why the distributed evaluation idea matters more than any individual result: it holds two of the three variables fixed across labs and forces the third into the open.
Does more data solve it?
Partly, and less cleanly than in language. Pooling helped, which Open X-Embodiment proved. But the pooled corpus was built from datasets collected for different projects on different bodies with different cameras, so a large part of the engineering was making them comparable at all. Adding another lab's logs adds coverage and adds heterogeneity in the same motion. That is a different scaling story from adding another billion web pages, where the format was already uniform.
There is also a quieter cost. Robot data ages. A dataset collected on hardware that has since been revised describes a machine that no longer exists in that configuration, and unlike text, the world it describes was the lab bench rather than the world.
What about safety and standards?
The rules moved recently, and less noisily than the demos. The Robot Report covered the overhaul of ISO 10218, announced as available on 18 February 2025 after work that took nearly eight years. The version it replaced dated from 2011.
Three changes matter for anyone thinking about deployment rather than research. Collaborative robot requirements that had lived in a separate technical specification were folded into the main standard. New robot classifications arrived with functional safety requirements and test methodologies attached, so the obligations scale with what the machine can do to a person. And cybersecurity requirements were added to an industrial safety standard, which is an admission that a networked robot has an attack surface as well as a reach envelope.
That last one connects robotics to a problem the software side already has. A physical system driven by a model that accepts instructions is a system where a bad instruction has kinetic consequences. If you want the software version of that argument, the pattern is the same one behind giving a model tool access through a defined protocol rather than an open channel: the interface is the control surface, and it is the only place you can enforce anything.
What would change the picture?
Four things, and the first credible instance of any of them is worth more attention than a hundred demo videos.
- A public success rate on novel objects, chosen by someone else. Not a curated set. Objects supplied by an outside party at evaluation time.
- Cycle times published next to success rates. The pair is the economic claim. Either number alone is a research claim.
- Deformable manipulation with a real error bar. Cloth and cable are the honest frontier. A policy that folds unfamiliar garments at a stated rate over a stated number of trials would be a genuine milestone.
- Distributed evaluation becoming normal. If pairwise cross-institution comparison becomes the default way policies are ranked, the field gets a shared yardstick it currently lacks.
Until then, the reasonable stance on ai robotics is neither dismissal nor excitement. It is a habit: when a clip appears, ask whether it was autonomous, whether it was one take, whether the object was new, how long it took, and how many attempts sit behind the number. Five questions, twenty seconds, and they will separate the results that matter from the ones that were filmed.