DexCanvas: Bridging Human Demonstrations and Robot Learning for Dexterous Manipulation

Xinyue Xu*, Jieqiang Sun*, Jing (Daisy) Dai*, Siyuan Chen, Lanjie Ma, Ke Sun, Bin Zhao, Jianbo Yuan, Sheng Yi, Haohua Zhu, Yiwen Lu

DexRobot Co. Ltd. · University of Michigan · Shanghai Jiao Tong University · Chongqing University · East China University of Science and Technology
*Equal contribution

arXiv GitHub HuggingFace

A large-scale hybrid real-synthetic dataset containing 7,000 hours of human manipulation data seeded from 70 hours of real demonstrations, organized across 21 fundamental manipulation types with physics-validated contact forces.

Dataset Overview

DexCanvas is a large-scale hybrid real-synthetic human manipulation dataset containing 7,000 hours of dexterous hand-object interactions seeded from 70 hours of real human demonstrations. Each entry combines synchronized multi-view RGB-D, high-precision mocap with MANO hand parameters, and per-frame contact points with physically consistent force profiles.

🎯

Contact Forces

Physics-validated per-frame contact points, force vectors, and object wrenches

📊

3.0B Frames

100× expansion from 70 hours of real human demonstrations

🤝

21 Types

Manipulation types from Cutkosky taxonomy

🎲

30 Objects

Geometric primitives + YCB objects

Data Processing Pipeline

Motion Capture Data

22-camera optical mocap system captures hand kinematics and object trajectories at 30 Hz

→

RL-based Force Reconstruction

Reinforcement learning policies reproduce demonstrations in physics simulation to extract contact forces

→

Physics-validated Annotations

Per-frame contact points, force vectors, and object wrenches measured from simulator

Manipulation Types & Objects

The taxonomy hierarchically organizes manipulation strategies into four main categories: Power (whole-hand grasps), Intermediate (transitional grasps), Precision (fingertip control), and In-hand Manipulation (dynamic reorientation).

Cutkosky Taxonomy of 21 Manipulation Types

Power Drill

Scissors

Banana

Bowl

Large Clamp

Bottle

Pitcher

Sphere 1

Sphere 2

Sphere 3

Cube 1

Cube 2

Cuboid 1

Cuboid 2

Cuboid 3

Cylinder 1

Cylinder 2

Cylinder 3

Cylinder 4

Cylinder 6

Join Us

DexCanvas represents our first step toward building foundation models for high-DOF dexterous manipulation. We're tackling the full complexity of 30+ degree-of-freedom hand-object interactions—transforming how robots learn to manipulate the physical world with human-level dexterity.

Research Scientist & PhD Internships

Working at the intersection of foundation models and embodied AI, you'll help shape how robots learn complex manipulation through large-scale learning from human demonstrations and physics-grounded simulation.

Students seeking research internships are especially welcome. Flexible duration (3-6 months).

Location & Contact

Shanghai, China

Contact: lyw@dex-robot.com

Learn more: dex-robot.com