Project demonstration footage shows a person bending to wipe a door and reaching for a higher spot. A moment later a humanoid (人形机器人) retargets the same sequence to its own shorter arms and stiffer joints, and a simulator checks whether it can stay upright. Nothing was filmed in a motion-capture studio. The source was an ordinary video found online.
That is the basic unit of Movens-5M, a dataset released in November 2026 by Zhejiang University, Unitree Robotics and their joint laboratory for embodied intelligence (具身智能), with partners including Microsoft Research, Peking University and the National University of Singapore.
What was actually released
Movens-5M contains 5 million robot-executable motion sequences and 15 million human-motion sequences, together spanning more than 10,000 hours. The actions cover 297 fine-grained types — walking, squatting, mopping, boxing and more — across 16 scenario categories such as household chores, fitness and outdoor leisure.
The unusual part is the alignment. Each clip is paired with text instructions, video demonstration, speech and music, plus time-segmented descriptions of what each body part is doing. Some samples carry time-stamped captions describing the action, alongside the recovered three-dimensional pose and the robot's simulated execution.
Alongside the data, the team trained Movens-Zero, an understanding–generation–execution architecture that takes all four input types and produces full-body motion on a Unitree G1. They also published Movens-Intent, a benchmark of 8,192 expert-verified condition–motion pairs across text, speech, music and video.
The paper appeared on arXiv (2610.41877) at the end of October 2026; the dataset, trained models and benchmark saw a wave of attention on 18 November. The dataset is hosted on Hugging Face, and the project has a public homepage.
How a cleaning video becomes robot training data
Converting web video into robot motions is not a download-and-go job. The team describes a five-stage automated pipeline.
Clip generation splits long videos along shot boundaries. Quality filtering drops material where the person is unclear or the motion information is thin. Semantic annotation adds scene and action descriptions, broken down by time segment and body part. Motion recovery estimates the human's three-dimensional pose frame by frame. Physical refinement then corrects jitter and instability, retargets the movement to the G1's proportions and joint limits, and runs it in simulation to verify that the robot can execute it without falling.
That last step is the one that makes the label "robot-executable" meaningful. A human motion is not automatically a robot motion: proportions differ, joint ranges differ, and balance has to be maintained. The simulated G1 tracking view is the check that the transformation worked.
It also defines the honest limit of the dataset. Movens-5M is verified in simulation on one robot body. Verification is not the same as reliable physical deployment.
One model, four kinds of instructions
Movens-Zero uses an understanding module to process text, speech, music or video, and a single "action expert" that combines that input with the robot's current state to plan motion. A controller then sends joint commands.
In demonstrations, the text prompt "Dance some hip-hop" produces a hip-hop dance without a reference video; a spoken "Salute with your left hand" produces a salute; music drives stepping and upper-body movement; and a video clip — from the internet or filmed live — is converted into a robot-feasible motion.
For live interaction, the model generates short chunks of motion. The robot begins executing the first part while the model keeps the unexecuted tail as the prefix for the next chunk, conditioned on the newest instruction and body state. That creates a continuous perception–generation–execution loop, which matters when a person changes their mind mid-demo.
The scaling curves are the real story
The headline number is five million sequences. The more informative result is what happens when the training set and the action expert grow.
Holding the largest action expert fixed at 910.5 million parameters, the team scaled training data from 1 percent of Movens-5M to 100 percent — roughly 122 hours to over 12,000 hours of deduplicated motion. Text-to-motion matching (R@3) rose from 12.72 percent to 62.57 percent. Video motion error (MPJPE) fell from 132.82 millimetres to 40.90 millimetres, a 69.2 percent reduction. Speech matching climbed from 13.98 percent to 74.30 percent, music beat alignment from 0.23 to 0.61, and tracking score from 0.21 to 0.72.
In a second experiment, they fixed the data and grew the action expert from 64.2 million to 910.5 million parameters. Validation flow-matching loss fell from 0.487 to 0.159, a 67.4 percent drop. In a separate experiment with a different baseline, tracking improved from 0.18 to 0.72. Larger experts also gained more from the same additional data: on text-to-motion matching, scaling data from 10 percent to 100 percent lifted the smallest expert from 6.34 percent to 10.30 percent, but the largest from 24.92 percent to 62.57 percent.
These are the authors' experiments, not an independent benchmark. Still, the direction is the point: for once in robotics, more data and a bigger model produced a measurable, monotonic improvement rather than a plateau.
Key takeaways
- Zhejiang University, Unitree Robotics and partners released Movens-5M: 5 million simulation-verified humanoid (人形机器人) motions and 15 million human motions, over 10,000 hours.
- The data was built from internet video through a five-stage pipeline that retargets human motion to a Unitree G1 and verifies it in simulation.
- Movens-Zero generates full-body motion from text, speech, music or video, with real-time closed-loop execution on the G1.
- Scaling data from 1 percent to 100 percent cut video motion error by 69.2 percent and more than quadrupled text-to-motion matching.
- A companion benchmark, Movens-Intent, contains 8,192 expert-verified pairs.
Honest limitations
Every performance figure above comes from the research team's own paper and scaling experiments. I found no independent replication of the results.
Simulation verification is not real-world reliability. The public demonstrations run on one robot platform, the Unitree G1, in controlled settings; the paper does not provide a success rate for long, unstructured physical tasks. Whether the motions transfer to other bodies is not established.
Building a dataset from internet video raises questions about source consent and licensing that the technical report does not fully resolve. The write-up does not spell out commercial use terms in detail, so developers should read the dataset licence before using it commercially. The dataset and models were also a very recent release, with limited community testing at the time of writing.
What readers can do now
- If you build embodied systems, download the dataset and check the licence first. The value here is in the paired modalities and the time-segmented descriptions, not just the raw motion count.
- If you evaluate robotics claims, separate simulated verification from physical deployment. "Executable in simulation" is a real contribution, but it is not a guarantee on a factory floor.
- Watch whether other labs reproduce the scaling curves. The open release makes that test possible, and it is the fastest way to learn whether the result is general.
