HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video

Accepted as Highlight (Top 11.9%)

Zicong Fan1,2, Maria Parelli1, Maria Eleni Kadoglou1, Muhammed Kocabas1,2, Xu Chen1,2, Michael J. Black2, Otmar Hilliges1
1ETH Zürich, Switzerland, 2Max Planck Institute for Intelligent Systems, Tübingen, Germany
Paper arXiv Video Code Contact

HOLD can reconstruct detailed 3D geometries of novel objects and hands in interaction from in-the-lab static 3rd-person videos and challenging in-the-wild 1st-person videos.

Abstract

Since humans interact with diverse objects every day, the holistic 3D capture of these interactions is important to understand and model human behaviour. However, most existing methods for hand-object reconstruction from RGB either assume pre-scanned object templates or heavily rely on limited 3D hand-object data, restricting their ability to scale and generalize to more unconstrained interaction settings. To this end, we introduce HOLD -- the first category-agnostic method that reconstructs an articulated hand and object jointly from a monocular interaction video. We develop a compositional articulated implicit model that can reconstruct disentangled 3D hand and object from 2D images. We also further incorporate hand-object constraints to improve hand-object poses and consequently the reconstruction quality. Our method does not rely on 3D hand-object annotations while outperforming fully-supervised baselines in both in-the-lab and challenging in-the-wild settings. Moreover, we qualitatively show its robustness in reconstructing from in-the-wild videos.

Video

BibTeX


@article{fan2024hold,
  title={{HOLD}: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video},
  author={Fan, Zicong and Parelli, Maria and Kadoglou, Maria Eleni and Kocabas, Muhammed and Chen, Xu and Black, Michael J and Hilliges, Otmar},
  booktitle = {Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year = {2024}
}