MacroHands: 3D hands and camera pose from egocentric video

MacroHands is a family of models that estimates 3D hand positions and camera motion from head-mounted video, aiming to make recordings of people doing tasks useful for training robots. Such video does not directly record robot actions, so hand motion must be tracked and then mapped to a robot’s gripper or hand. Existing trackers can fail when hands are occluded, gloves are worn, or other people or body parts appear; inaccurate depth can also place learned actions far from their true positions. MacroHands-Mono uses one camera, while MacroHands-Stereo uses both cameras in a stereo headset. Both track the wearer’s hands through occlusion and estimate the camera’s path, allowing hand motion to be represented in the surrounding world.

The reported evaluations compare the models with open-source baselines on ARCTIC, HOT3D, and SHOW3D, using captured ground truth for hand joints and camera pose. On SHOW3D, which no compared model had trained on, MacroHands-Mono’s unaligned 3D hand-joint error is 32.2 mm, versus 55.9 mm for HaWoR; stereo reduces it to 13.1 mm, mainly by improving depth. On partly occluded clips, the system finds 80% of hands, and only 0.2% of its reported hands land on someone other than the wearer. With stereo and headset motion sensors, its camera-path drift is substantially lower than the cited SLAM baselines. On HOT3D, its next-second hand-motion error in world coordinates is 13.6 mm, compared with 65.9 mm for HaWoR. These results matter because robot training depends on continuous, correctly located motion rather than hand shape alone.