Manifest Labor
Multi-modal motion capture pipeline for skilled trade demonstrations, structured for humanoid robot training. Targeting the gap between lab-demo training data and real-world construction work.
Context
Most humanoid robot training data comes from lab demonstrations and simulation. Skilled trade work (welding, pipefitting, fabrication) is largely absent from existing datasets like RT-X, Open X-Embodiment, and DROID. The hand movements, force profiles, and real-time adjustments that experienced tradespeople rely on are embodied knowledge that hasn't been captured in a format usable for ML training.
At the same time, the construction industry faces a growing labor shortage, and the skilled trades workforce is aging out.
Approach
Manifest Labor is a data capture pipeline for skilled trade demonstrations. The system records multi-modal data (video, motion capture, force sensors, audio) and structures it for ML training from the point of collection. The plan is to partner with trade unions, vocational schools, and retired masters to record actual work rather than staged demonstrations.
The guiding research question: what is the smallest dataset that validates the concept? The current focus is welding, chosen for its complex motion profiles and strong automation demand. The pipeline architecture is trade-agnostic.
Scope
| Parameter | Value |
|---|---|
| Target trade (MVP) | Welding (TIG/MIG) |
| Capture modalities | Video (multi-angle), IMU motion capture, force/torque sensors, audio |
| Benchmarked datasets | RT-X, Open X-Embodiment, DROID |
| Partner targets | Trade unions, vocational schools, retired journeymen |
| Output format | ML-ready structured data, temporal alignment across modalities |
Direction
Near-term application is construction automation, where the labor shortage creates immediate demand. Longer-term, this work is relevant to pre-human infrastructure deployment: robotic systems that can fabricate and assemble structures in environments like Mars before crewed missions arrive. That requires training data grounded in real trade expertise.
The project is in early exploration and validation. The priority right now is scoping the minimum viable capture rig and locking the MVP trade.