Can Robots Learn by Watching People?
Teaching a robot a new task often requires engineers to program its movements or collect demonstrations using the robot itself. An alternative is to let the robot observe a person performing the task and learn from the demonstration.
Known as imitation learning, this approach could simplify robot training in manufacturing, warehouses and healthcare support. However, observing an action is not the same as understanding every movement and physical interaction involved. Research published in 2026 is exploring how robots can learn from human videos with fewer robot-specific demonstrations.
Motion Tracking: Turning Human Actions into Data
Cameras and computer vision systems estimate the positions of hands, joints and objects over time. Some approaches also use depth cameras or wearable sensors to capture three-dimensional movement.
For example, when a worker picks up a component, rotates it and inserts it into a machine, the system must track the hand, identify the object and recognise important interactions.
Human movements cannot simply be copied joint by joint. People have flexible fingers and different body proportions, while robots may use rigid joints or simple grippers. This difference, known as the embodiment gap, means human movements must be translated into actions suited to the robot's physical design.
Imitation Learning: From Demonstration to Robot Action
After processing a demonstration, a learning model can develop a policy—a system that selects robot actions based on visual information and task requirements.
A 2026 study introduced Human2Robot, which learns from paired human and robot videos. Researchers created 2,600 synchronised episodes using a virtual-reality (VR) teleoperation system. The dataset covered four basic tasks and six longer, multi-step tasks. Experiments examined whether learned skills transferred to new objects, positions and task categories. These results demonstrate potential within the tested settings, not universal performance across robots.
Another study, ORION, explored learning manipulation skills from a single human video. It tracked objects and key points to create a sequence of interactions. Researchers reported a 74.4% average success rate across their real-world evaluations, covering short- and long-horizon tasks and RGB and RGB-D inputs. This result applies to the study's tested conditions, not all robots or tasks.
Skill Transfer: Learning Connected Actions
Complex tasks require several coordinated actions. Assembling a product, for example, may involve picking up a component, aligning it, fastening it and checking the result.
In June 2026, the University of Maryland reported on HumanEgo, a framework exploring how robots can learn object manipulation from first-person human videos without robot demonstrations or robot-specific training data.
Another study, CIVIL, used physical markers and natural-language prompts during training to help robots identify task-relevant objects. Simulation and real-world experiments examined its learning efficiency and generalisation.
These approaches show that transferring skills involves recognising relevant objects, understanding action sequences and responding to changing surroundings—not merely copying movements.
Why Copying Human Movements Is Still Difficult
A demonstration cannot cover every situation. Lighting may change, objects may slip or tools may be positioned differently. A robot that memorises one trajectory may fail when conditions change.
Reliable systems need visual feedback, force control and the ability to adjust actions. Human videos may also hide contact forces or important details behind hands and tools, making interactions difficult to interpret.
Safety matters too. Robots must avoid collisions, excessive force and damage to equipment. Real-world deployment requires task-specific testing, appropriate safety constraints and human supervision where necessary.
What Comes Next?
Learning from human demonstrations could make robot training faster and more flexible, especially in workplaces where products and processes frequently change. However, demonstrations provide a starting point, not a guarantee of success. Combining observation with feedback, physical reasoning and repeated testing will be essential for robots to adapt learned skills safely to unfamiliar situations.
