Hand-motion data for embodied AI should place environmental observations, human movement, contact state and task outcomes on a common timeline. Video or joint angles alone rarely explain why an object slipped, when a fingertip first touched it or why an insertion succeeded. A useful dataset helps the target robot reproduce a task and helps engineers understand failures.
Research such as ACT and Diffusion Policy shows how real demonstrations and action sequences can support learned manipulation policies.[1][2] Those experiments do not guarantee results from an arbitrary acquisition device. A dataset still needs clearly defined tasks, hardware, action representations and evaluation conditions.
Define the task before choosing the channels
Picking up a paper cup, twisting a cap and inserting a connector place different demands on data. A cup task emphasizes contact position and grip changes; a cap involves finger coordination, rotation and friction; insertion requires alignment, contact timing and corrective motion. Define success, acceptable error, object coverage and constraints before selecting sensors and sampling rates.
Retain the task instruction, object and material information, operator ID, device and calibration versions, start and end times, and outcome for each demonstration. For images and behavior involving people, establish collection consent, retention limits and access permissions. This context makes a recording interpretable and easier to filter or reuse.
Vision, motion and touch answer different questions
Vision describes appearance, scene context and relative position. Motion describes changes in joints, wrists or fingertips. Touch adds contact regions and force changes. Similar joint angles can occur while a hand is in free space, lightly touching an object or gripping it firmly. Distinguishing these states explains the interaction rather than simply describing hand shape.
Keep measurements, algorithmic estimates and control commands distinct. A commanded joint target is not the measured joint state; camera-estimated landmarks are not force measurements. Document each field's units, coordinate frame, origin, validity and confidence. Missing channels should remain explicitly missing rather than being filled with plausible-looking zeros.
Synchronize events, not just frame counts
Cameras, motion systems and tactile arrays often update at different rates. Resampling them to one frame rate does not remove latency. Preserve acquisition and receive timestamps, document the clock source, synchronization method, transport delay and interpolation rules, then test alignment with events visible across channels.
When a fingertip touches an object, visible deformation, rising tactile signals and changing motion should have a sensible temporal relationship. File write order can obscure that relationship when a network is congested. Acceptance checks should measure time offsets, drift, dropped frames and consecutive gaps, alongside nominal sampling rates.
Retarget human motion to the robot
Human hands and robotic hands differ in anatomy, joint count and range of motion. Copying angles directly may create unreachable poses or move fingertips away from an object. Retargeting should preserve task-relevant constraints: relative fingertip positions, contact order, palm orientation, joint limits and collision constraints. Save mapping parameters with the data.
Coordinate conventions matter just as much. ROS REP 103 provides a reference for SI units and right-handed frames, including the distinction between body and camera optical frames.[4] Other conventions are possible, but transformations and rotation representations must be documented. Mirroring, mixed angle units and reversed axes can otherwise create difficult-to-diagnose training errors.
Keep contact phases and informative failures
Consider labeling approach, initial contact, grip establishment, manipulation and release, together with slip, collision, regrasp and correction events. Failed demonstrations can be useful when their causes and permitted uses are clear. An unlabeled failure should not become positive action supervision by default, while edited-out pauses and corrections may contain valuable task information.
For insertion, distinguish misalignment before contact, obstruction after contact and loss of grip during manipulation. All may end in failure, but their observations and action causes differ. Event definitions, annotation examples and agreement checks provide more value than a single success/failure flag.
Validate with independent tasks and conditions
Report usable demonstration time, task coverage, object variety and complete contact episodes—not only frame counts. Randomly splitting adjacent video frames across training and test sets can inflate scores because the images are nearly identical. Split by complete demonstration, operator, object or environment, with explicitly held-out conditions and fixed rules.
Open X-Embodiment shows the potential of collecting robot data in standardized formats for learning across platforms.[3] In a project, format alignment is only the beginning: action semantics, hardware differences and timing still need attention. Establish reproducible baseline tasks and compare added data under the same conditions to identify where improvements come from.
Two questions before collection starts
How many demonstrations are enough? There is no universal number. Cover the critical contact phases and main variations first, build a small training and evaluation baseline, then identify the conditions in which it still fails. Repeating similar motions with one object and one operator may add frames without expanding coverage. Prioritize the gaps exposed by failures.
Can touch be added later to motion-only recordings? New demonstrations can include tactile sensing, but the original contact generally cannot be recovered losslessly from pose. Signals collected in separate interactions are not automatically frame-aligned pairs. Decide whether future work will study contact control before selecting the initial raw channels. For existing data, state which tasks it can support and what information can realistically be added.
An acquisition system should deliver usable data
Ensuring Magic Glove integrates hand-motion and tactile acquisition in one system. Evaluation should consider fit across hand shapes, natural manipulation while wearing the device, calibration repeatability, usable data yield and export documentation. Sensor array count describes the acquisition configuration; it does not establish task success.
A handover package should include raw data, field definitions, calibration files, synchronization records, task labels and a reproducible reading example. When a team can replay recordings, investigate anomalies and repeat evaluation with the same tools, collection becomes a source of trainable, verifiable physical interaction data.
References
Publications and technical documentation cited in this article. Last checked: 2026-10-19.
- Zhao et al. — Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (2023) ACT: learning fine-grained bimanual manipulation from demonstration sequences.
- Chi et al. — Diffusion Policy (2023; revised 2024) Visuomotor policy learning through action distribution modeling.
- Open X-Embodiment Collaboration — Robotic Learning Datasets and RT-X Models (2023) Standardized robot datasets and cross-embodiment learning.
- ROS REP 103 — Standard Units of Measure and Coordinate Conventions SI units, right-handed coordinate systems, axes and rotation representations.