Learning to Disambiguate Strongly Interacting Hands via Probabilistic Per-Pixel Part Segmentation

Oral Presentation, 3DV 2021

1ETH Zürich, 2Max Planck Institute for Intelligent Systems, Tübingen

DIGIT is robust to appearance ambiguities when hands interact. Here both methods use a soft-argmax formulation.


In natural conversation and interaction, our hands often overlap or are in contact with each other. Due to the homogeneous appearance of hands, this makes estimating the 3D pose of interacting hands from images difficult. In this paper we demonstrate that self-similarity, and the resulting ambiguities in assigning pixel observations to the respective hands and their parts, is a major cause of the final 3D pose error.

Motivated by this insight, we propose DIGIT, a novel method for estimating the 3D poses of two interacting hands from a single monocular image. The method consists of two interwoven branches that process the input imagery into a per-pixel semantic part segmentation mask and a visual feature volume. In contrast to prior work, we do not decouple the segmentation from the pose estimation stage, but rather leverage the per-pixel probabilities directly in the downstream pose estimation task. To do so, the part probabilities are merged with the visual features and processed via fully-convolutional layers.

We experimentally show that the proposed approach achieves new state-of-the-art performance on the InterHand2.6M dataset. We provide detailed ablation studies to demonstrate the efficacy of our method and to provide insights into how the modelling of pixel ownership affects 3D hand pose estimation.



Problem Appearance ambiguities in strongly interacting hands.

Solution Per-pixel part segmentation provides semantics for each pixel, allowing downstream pose estimation to disambiguate different parts of the two hands.

Method Overview

An illustration of our hand pose estimation model (DIGIT). Given an image, DIGIT extracts visual features (F) and predict part segmentation probability volume (S). The segmentation volume is projected into latent semantic features (S'). The visual features (F) and the semantic features (S') are fused across multiple scales and are used for interacting hand pose estimation.



  title={Learning to Disambiguate Strongly Interacting Hands via Probabilistic Per-pixel Part Segmentation},
  author={Fan, Zicong and Spurr, Adrian and Kocabas, Muhammed and Tang, Siyu and Black, Michael and Hilliges, Otmar},
  booktitle={International Conference on 3D Vision (3DV)},