KODA

Kinematics–Observation Data Alignment for Humanoid Learning from Human Egocentric Demonstration

Zibo Zhou^1,2,3† Yueru Chen^2,3† Junsong Wu^2,3 Weiji Xie^2,3 Qingyao Xu³ Sheng Yin³ Shiqing Wang^2,4 Jiyuan Shi^1,2 Siheng Chen³ Chenjia Bai^1,2* Xuelong Li^2*

¹Gamma Robotics ²Institute of Artificial Intelligence (TeleAI), China Telecom ³Shanghai Jiao Tong University ⁴Zhejiang University

^†Equal contribution (co-first authors) ^*Corresponding authors

Video Code PDF Arxiv

01 / Abstract

Turn egocentric demonstrations into controller-ready whole-body behavior.

Overview of KODA, from egocentric human data through whole-body motion and robot onboard view generation to controller-specific VLA training data — Figure 1 Overview of KODA. Whole-Body Motion Generation produces simulation-feasible whole-body motion from human egocentric demonstrations, and Robot Onboard View Generation synthesizes motion-aligned onboard views. The motion, views, and task instruction form a reusable canonical record. Control Command Generation uses this record to generate controller-specific states and actions. The resulting data are combined with matching real-robot demonstrations to co-train a controller-specific vision–language–action (VLA) policy for language-guided humanoid loco-manipulation.

KODA closes the interface gap between what a person sees and does during a demonstration and what a humanoid controller needs at training time.

We align human visual observations, head and hand motion cues, robot states, compatible actions, and task language on one control clock. The result is a reusable data conversion layer that can supplement different target controllers without rebuilding the model prior from scratch.

02 / Method

A four-step conversion path keeps the data traceable.

The central design choice is to make the conversion explicit: every aligned record can be inspected from source observation through controller-specific actions.

KODA pipeline from human data through whole-body motion, robot onboard view generation, and controller-specific VLA data

Figure 2 KODA pipeline. Middle: Human demonstrations become aligned whole-body motion and onboard observations, then controller-specific state–action data for VLA training (Control Command Generation). Top: Whole-Body Motion Generation combines hand estimation and contact-aware retargeting, sampling-based command optimization (SCO), and arm refinement. Bottom: Robot Onboard View Generation combines arm removal, scene reprojection, robot rendering, depth compositing, and inpainting. Input depth is observed or estimated from RGB.

01 · CAPTURE

Observe the task

Record the human viewpoint, hand and head cues, and language goal with the demonstration.

02 · CONVERT

Recover whole-body motion

Map visual motion into robot motion and robot views while preserving the original task context.

03 · ALIGN

Instantiate the interface

Project states and actions onto the target controller’s compatible action space and control clock.

04 · TRAIN

Fine-tune the policy

Combine KODA records with robot demonstrations to train a controller-specific whole-body VLA.

Every training record keeps robot observations, whole-body states, compatible actions, and task instruction together so later debugging can follow the same path as training.

03 / Video

A short film for the KODA story.

This paper film introduces the motivation, conversion path, and controller-specific supervision behind KODA.

KODA · paper film3:04 · 1920 × 1080

04 / Results

Browse the data demos and controller rollouts.

The short-form library is grouped by role: data processing clips, canonical KODA records, warehouse logistics demonstrations, and successful rollouts from the SONIC and HEFT whole-body controllers.

Data demo 01 · Processing clip

Short visualization of the KODA data processing and alignment pipeline.

Data demo 02 · Processing clip

Short visualization of the KODA data processing and alignment pipeline.

Data demo 03 · Processing clip

Short visualization of the KODA data processing and alignment pipeline.

Data demo 04 · Processing clip

Short visualization of the KODA data processing and alignment pipeline.

Data demo 05 · Processing clip

Short visualization of the KODA data processing and alignment pipeline.

Data demo 06 · Processing clip

Short visualization of the KODA data processing and alignment pipeline.

Data demo 07 · Processing clip

Short visualization of the KODA data processing and alignment pipeline.

Data demo 08 · Processing clip

Short visualization of the KODA data processing and alignment pipeline.

KODA canonical · Task 01 · Pick and place

Canonical KODA record for whole-body reach and placement.

KODA canonical · Task 02 · Container transfer

Canonical KODA record for object transfer with synchronized hand and robot views.

KODA canonical · Task 03 · Surface interaction

Canonical KODA record for contact-rich action grounded in a controller-compatible trajectory.

KODA canonical · Task 04 · Rearrangement

Canonical KODA record for multi-step rearrangement with the task instruction retained.

SONIC rollout · Task 01 · Pick and place

Successful whole-body controller rollout for whole-body reach and placement.

SONIC rollout · Task 02 · Container transfer

Successful whole-body controller rollout for object transfer with synchronized hand and robot views.

SONIC rollout · Task 03 · Surface interaction

Successful whole-body controller rollout for contact-rich action grounded in a controller-compatible trajectory.

SONIC rollout · Task 04 · Rearrangement

Successful whole-body controller rollout for multi-step rearrangement with the task instruction retained.

HEFT rollout · Task 01 · Pick and place

Successful whole-body controller rollout for whole-body reach and placement.

HEFT rollout · Task 02 · Container transfer

Successful whole-body controller rollout for object transfer with synchronized hand and robot views.

HEFT rollout · Task 03 · Surface interaction

Successful whole-body controller rollout for contact-rich action grounded in a controller-compatible trajectory.

HEFT rollout · Task 04 · Rearrangement

Successful whole-body controller rollout for multi-step rearrangement with the task instruction retained.

Logistics demo 01 · Autonomous bin handling

Autonomous humanoid manipulation in a warehouse-style logistics setting.

Logistics demo 02 · Autonomous cart transfer

Autonomous humanoid logistics demonstration with cart interaction and transfer.

05 / Experiments

Human demonstrations expand whole-body policy coverage.

KODA is evaluated on a Unitree G1 with BrainCo hands across four loco-manipulation tasks. The same canonical records are reused to train both SONIC and HEFT policies with fixed robot data.

+45.0 ppSONIC Robot-OOD3.75% → 48.75% with 100 KODA demos

+46.25 ppHEFT Robot-OOD5.00% → 51.25% with 100 KODA demos

4Whole-body tasksCup, parcel, basket, and drawer tasks

20Real-robot trialsPer task and condition

Robot-OOD success scales with converted human data

Macro-average task success under conditions covered by human demonstrations but absent from the robot demonstrations.

SONIC · 3.75 → 48.75%HEFT · 5.00 → 51.25%

KODA demos / task	SONIC OOD	HEFT OOD	SONIC ID	HEFT ID
0	3.75%	5.00%	62.50%	68.75%
25	16.25%	20.00%	68.75%	66.25%
50	23.75%	31.25%	70.00%	68.75%
75	41.25%	45.00%	71.25%	72.50%
100	48.75%	51.25%	72.50%	76.25%

Observation alignment

SONIC Robot-OOD macro-average success across the four alignment variants.

No alignment18.75%

Fixed offset23.75%

Motion reprojection37.50%

Full alignment48.75%

Variant	Robot-ID	Robot-OOD
No alignment	62.50%	18.75%
Fixed offset	63.75%	23.75%
Motion reprojection	66.25%	37.50%
Full alignment	72.50%	48.75%

Controller-specific command optimization

Sampling-based command optimization (SCO) improves both in-distribution and out-of-distribution execution.

48.75 → 72.50%Robot-ID macro-average

7.50 → 48.75%Robot-OOD macro-average

+41.25 ppRobot-OOD gain

Without SCO, canonical motion is used directly as the tracking reference. With SCO, the controller-specific commands are optimized before recording state-action supervision.

Motion fidelity and controller trackability

Lower is better. Position is reported in centimeters and orientation in radians.

Stage / target	Direct	SCO	SCO + IK
Whole-body generation · Wrist position / orientation	15.02 cm / 0.2046 rad	3.03 cm / 0.1106 rad	0.33 cm / 0.0296 rad
SONIC rollout · Wrist position / orientation	61.50 cm / 0.1777 rad	2.88 cm / 0.1292 rad	—
HEFT rollout · Wrist position / orientation	19.61 cm / 0.0677 rad	2.94 cm / 0.0552 rad	—

Experimental protocol from the paper: 50 robot demonstrations and up to 100 converted human demonstrations per task; one training run per configuration; 20 real-robot trials per task and condition; macro-averages weight the four tasks equally.

06 / Dataset

Each record carries the signals a whole-body policy needs.

KODA makes the data processing chain visible: source demonstrations become aligned records, then controller-specific training data.

An aligned KODA training record contains robot observations, whole-body states, compatible actions, and task instruction — FIG. 02 · One training record is paired on a single control clock.

01

Robot observations

Onboard RGB and the visual context available to the target policy.

02

Whole-body states

Body joints, gravity, and hand state expressed in the target embodiment.

03

Compatible actions

Commands projected into the controller’s action space for execution.

04

Task instruction

The language goal that keeps the record usable for instruction following.

Egocentric RGBHuman viewpoint

Head + handsMotion cues

Robot stateWhole-body status

Action tokensController commands

InstructionLanguage goal

Control clockAligned timestamps

07 / Citation

Keep the project easy to reference.

Citation details will be updated once the paper metadata is final.

@article{koda2026,
  title   = {KODA: Kinematics--Observation Data Alignment for Humanoid Learning},
  author  = {Zibo Zhou and Yueru Chen and Junsong Wu and Weiji Xie and Qingyao Xu and Sheng Yin and Jiyuan Shi and Siheng Chen and Chenjia Bai},
  year    = {2026},
  note    = {Project page}
}