Estimating 3D Hand Poses and Shapes from Silhouettes

Hundreds of years ago, the historic art form of hand shadow puppetry was already documented in many ancient countries. Since then, making hand shadow puppets has been a low-cost household entertainment activity that parents can play with their children. Regarding human perception, it is interesting that humans can recognize the animals and characters portrayed by the hands and fingers from merely their projected shadows. It is also remarkable that people can develop and create different ways of making hand shadow puppets, which may involve the generative capability of imagining some target shape and adjusting the hands and fingers to match the imaginary shape with the shadow. Inspired by the discriminative and generative nature of the visual process of making hand shadow puppets, we aim to explore the possibility of learning to estimate the 3D hand pose and shape from an input hand silhouette through a similar trial-and-error manner.

Our primary goal in this endeavor is to train a renderer-equipped deep network that takes a binary mask of a hand’s silhouette as the input and predicts the corresponding 3D hand shape that produces a very similar silhouette. (Figure 1 shows several example results obtained using our model.) To build such a model, we employ MANO [25] as the 3D hand-shape renderer in our deep network. We propose a new network called Mask2Hand, which consists of an encoder, a MANO layer, a differentiable render, and the refinement module. Figure 2 shows an overview of the proposed Mask2Hand network. Our network takes a binary mask as the input and uses the encoder to generate the required parameters for the MANO layer to render the hand mesh. The encoder learns to predict the pose-related principal components of MANO and the 3D global transformation of the hand from only a single input binary mask. Based on the principal components and global transformation, the MANO layer reconstructs the 3D hand mesh and the hand joint positions. We then use a differentiable renderer to generate the 2D projected silhouette from the hand mesh. Finally, the refinement module compares the projected silhouette with the input binary mask to refine the entire network. Three groups of loss functions are tailored for training the aforementioned modules, including pose loss, silhouette loss, and mesh loss. The loss functions and the operations involved in the modules are all differentiable—we are able to train the entire network from end to end. Our code can be found on GitHub by searching Mask2Hand.

As this work aims at addressing a rather new and challenging task of predicting the 3D hand pose from a single 2D binary mask, it is quite encouraging that the proposed Mask2Hand network can achieve comparable prediction accuracy on 3D hand pose and shape estimation as state-of-the-art methods, although we only need a single binary input image while theirs require an RGB image or a depth map. Note that our model can be trained under supervision with manually annotated joint positions or fully self-supervised without human annotations. Furthermore, since our method learns to predict the global transformation of the hand, its evaluation performance under the unaligned setting of hand pose estimation does not degrade as much as other methods do. Such a property is particularly favorable as, in real applications, it is not allowed to align the prediction with the unknown ground truth.

In addition to the motivation of hand shadow puppets, we would like to point out several scenarios of real-world applications in which working on binary silhouettes has advantages.

1. Accessibility with inexpensive low-resolution sensors: Unlike RGB input, binary images can be obtained with relatively inexpensive sensors, such as low-resolution range and infrared sensors. They are especially useful when the lighting condition is poor and not much color information is available.

2. Potential for ubiquitous sensing: A line of research [17, 21, 31] in the hardware domain explores the usage of ambient light as a ubiquitous sensing medium. Light sensors, such as photodiodes or solar cells, measure and track the shadow blockage that the user’s hands cause with low cost and ultra-low power consumption, which have the potential to be widely used in IoT devices for shadow-based hand pose estimation systems.

3. Robustness to adversarial attacks: Binary input is more robust to adversarial attacks of imperceptible pixel perturbation. From the literature on adversarial machine learning [1], we know that invisible image manipulation can seriously spoil the functionality of DNNs. However, this common type of attack is not directly applicable to the binary setting since small perturbation in pixel values is impossible with binary images. Also, pixel inversion in binary silhouettes can be easily identified and addressed.

One of the research endeavors inspiring us is Shadow Theatre [33], which aims to reproduce the shadow puppet theatre computationally—an ancient shadow-oriented form of performance art—just like how we are motivated by hand shadow puppetry. Won and Lee [33] develop an algorithm to incorporate several features and heuristic rules that are related to shadow generation using human bodies. The proposed method optimizes the poses of human bodies to match the given 2D target shapes. The underlying nonlinear high-dimensional optimization is solved using the proposed heuristic strategies to find a plausible solution with reasonable computing resources. Nevertheless, the computation still requires tens to hundreds of minutes to complete on GPU.

Various methods have been proposed to address the problem of 3D hand pose and shape estimation. Generally, the techniques that are used to solve 3D human pose estimation can also be applied to 3D hand pose estimation. Based on the input format, we consider the following four categories of methods for 3D hand pose and shape estimation: RGB-based, depth-based, event-camera-based, and silhouette-based methods. Among them, silhouette-based methods are most closely related to this work, where we seek to generate the 3D hand pose and mesh from merely a single binary-valued image of a hand silhouette.

Despite the discrepancy in their input type, many of the methods take 2D joint detection as the basis. Such an approach is promising, and many existing deep-learning-based techniques can be employed. The following is a brief review of recent methods in the four categories.

RGB-based Methods. Lin et al. propose a method called Mesh Transformer (METRO) [19] that explores non-local relationships between vertices and joints to reconstruct a 3D human pose from a single RGB image without using parametric mesh models. This method can also be extended to hand mesh reconstruction. Chen et al. [5] propose Camera-space Mesh Recovery (CMR) that generates 3D meshes from a single RGB image by extracting 2D cues, including joint landmarks and silhouettes. Chen et al. [6] present a self-supervised 3D hand reconstruction network called S²HAND that estimates 3D poses from 2D detected keypoints. Liu et al. [20] propose a semi-supervised 3D hand-object pose estimation method leveraging spatial-temporal consistency in videos to obtain pseudolabels for self-training. Li et al. [16] propose a hand pose estimation method by using multi-task learning to categorize joints into groups so that different features can be learned to recover the 3D joint locations in a group-wise manner.

Depth-based Methods. Xiong et al. [34] propose the Anchor-to-Joint Regression Network (A2J) that exploits informative anchor points to enhance the generalization ability of 3D pose estimation. Fang et al. [9] present a dense-prediction-based 3D hand pose estimation method called JGR-P2O that uses a pixel-to-offset prediction network and a joint graph reasoning module to enable end-to-end training and improve computational efficiency.

Silhouette-based Methods. Silhouette-based methods have not yet been widely applied to hand/body pose estimation. The task to be solved is more challenging than the settings of RGB-based and depth-based methods. The difficulty comes from the lack of information in the input data, where the shading, color, and depth of the 3D objects are absent in the binary-valued images. Lee et al. present the Silhouette-Net [15] method that generates 3D hand poses based on binary-valued silhouettes and can achieve a performance similar to or better than depth-based methods if multiview silhouettes are provided. However, their method requires the guidance of depth maps during the training stage. In contrast, our method does not require additional depth information during training; it only needs a single-view 2D binary mask of the hand as the input. Despite the challenging setting, estimating the 3D hand pose and mesh from limited binary information in the input is our major goal in this work.

Several hand mesh models have been proposed, such as MANO [25], Sphere-Meshes [29], and Convex Parts [30]. With the help of these models, hand shape generation and pose estimation can be processed via parametric methods. Typically, a hand model like MANO is designed to parameterize a triangle mesh into pose, shape, and rotation parameters. MANO has been widely used by hand pose estimation methods [3, 2, 7, 8, 12, 23, 27, 32, 35, 36] to train their deep networks; the deep networks learn to predict the hand shape in the target image by regressing to the principal components of the MANO model. As shown in Figure 2, our method also uses the MANO model to construct the target hand shape. By learning to optimize the parameters of MANO meshes, our method achieves the goal of generating the 3D hand mesh from a binary-valued silhouette image.

This paper is an extended version of our previous conference paper, “Mask2Hand: Learning to Predict the 3D Hand Pose and Shape from Shadow” [4]. Mask2Hand is a self-trainable method that accurately predicts 3D hand pose and shape from simple 2D binary silhouettes; it offers significant advantages by eliminating the need for complex RGB or depth data. In this paper, we have substantially revised the conference version with the following enhancements:

• Detailed Model Description: We provide a thorough explanation of the model architecture to facilitate implementation and reproducibility.

• Enhanced Visualizations: We include additional qualitative results to better illustrate the effectiveness of the method.

• Robustness Analysis: A new section explores the impact of noisy input on our model’s performance.

• Limitations Discussion: We address the limitations of our approach.

To predict the hand pose and shape from a binary mask, we propose a two-phase end-to-end trainable network, Mask2Hand, that aims to reconstruct 3D hand meshes in the camera space. Figure 2 illustrates the overall pipeline of our method. In what follows, we detail each pipeline module and describe how we formulate the loss functions.

The core modules of our model include an encoder for estimating the MANO parameters, a MANO layer that reconstructs the 3D hand mesh, a differentiable renderer for optimizing the hand pose, and a RefineNet improving the accuracy.

Given a hand silhouette image presented as a binary mask, the encoder first uses a ResNet18 [13] backbone to extract features from the input. Then, these features are fed into four different estimators to encode 3D information into the PCA space of MANO [25] and predict the extrinsic camera parameters. More specifically, the output of the encoder consists of the first six or forty-five hand-pose-related principal components θ ∈ ℝ⁶ or ℝ⁴⁵ of MANO, shape parameters β ∈ ℝ¹⁰, 3D rotation R ∈ ℝ³ in axis-angle representation, and 3D translation T ∈ ℝ³. The detailed model architecture for our Encoder network is illustrated in Table 1.

The 3D hand shape is represented by a triangle mesh that contains a vertex set V ∈ ℝ^778×3 and fixed faces F indicating the connection between them. In our network, the hand mesh and joints J ∈ ℝ^21×3 are reconstructed using MANO [25]. Being a low-dimensional parametric hand model, MANO can generate the required hand mesh from parameters including the hand-pose-related principal components θ, shape β ∈ ℝ¹⁰, rotation R, and translation T.

Given intrinsic camera parameters and hand meshes in the camera space, we use the differentiable rendering technique [14] to project 3D estimations into the 2D image space. By applying various losses between the rendered mask of the hand silhouette and the input binary image, we can integrate self-supervision into our end-to-end optimization process for constraining global mesh registration. Here, we use the PyTorch3D [24] implementation of the differentiable renderer. The loss functions will be detailed later in the subsequent sections.

In the previous stage, we only consider several pose-related principal components of MANO for parametrizing hands in a low-dimensional space. The lower degree of freedom may have an expressiveness problem in that the finer details of hand gestures cannot be ideally reconstructed. Therefore, we incorporate a small convolutional neural network called the RefineNet module,¹ into our model to deal with this issue. The network takes the concatenation of the input image and the projected silhouette as input, and it produces residual coordinate values that serve as the point-wise offsets to the previously constructed mesh. The final output of our model is the summation of the originally reconstructed hand mesh and the offset predicted by the RefineNet. The refined joints are then estimated by MANO’s joint regressor, given the refined mesh as its input.

The complete objective function that we design for training our Mask2Hand model comprises three main loss terms:

where we have the pose loss ℒ_pose on the predicted joints, the silhouette loss ℒ_silhouette on the 2D projected silhouette of the hand, and the mesh loss ℒ_mesh on the reconstructed 3D mesh.

The pose loss in (1) further consists of two loss functions that use the unaligned and aligned joint positions to evaluate the preliminary and refined joint predictions by the squared L2-norm error:

where we have ℒ_J for the unaligned joint evaluation and ℒ_alignJ for the aligned joint evaluation, with λ_J and λ_alignJ as the weight factors. The 3D positions of the K joints are represented as a K by 3 matrix, i.e., J ∈ ℝ^K×3. The loss function ℒ_J of the unaligned joint evaluation is computed as

where J_i is the ground-truth 3D position of joint i, Ĵ_i is the preliminarily reconstructed position of joint i, and $Jiref ^$ is the final prediction after refinement.

For the aligned joint loss, the predicted joints are aligned with the ground truth using orthogonal Procrustes Analysis (PA) [26], and then the L2 loss is applied to the aligned and the ground-truth joints:

The procedure of alignment PA(J, Ĵ) can be described as follows.

1. Translate the joints so that their mean lies at the origin:

2. Divide the joints set by its Frobenius norm to remove the uniform scaling component:

3. Solve the orthogonal Procrustes problem: given matrices J and Ĵ, find an orthogonal matrix Q ∈ ℝ^3×3 that maps J most closely to Ĵ, i.e., $minQ‖JQ−J^‖F$ subject to Q^⊤Q = I_3×3. The solution consists of Q = UV^⊤ and $s=∑iσi$⁠, where UΣV^⊤ is the singular value decomposition of J^⊤Ĵ and σ_i’s are the singular values.

4. $J˜=(J^Q⊤×s)×‖J‖F+(∑i=1KJi)/K$⁠, where the Procrustes output is scaled by the norm and shifted by the mean that we derive from J in the first two steps.

The ideal result for the module of differentiable rendering is that the 2D projected hand silhouette matches the input binary mask exactly. Hence, our silhouette loss is based on the binary cross-entropy (BCE) and the contour loss:

where S is the rendered silhouette and B is the input binary mask. Our contour loss is a differentiable estimation of Chamfer distance between two contours, calculated as a pixel-wise multiplication between the contour of the rendered silhouette and the pre-computed distance field of the ground-truth silhouette’s contour. The detailed steps for computing the Chamfer distance are summarized as follows:

1. Compute the distance transform [10] for the contour ∂B of the ground-truth binary mask B in the dataset to get the distance field Ψ_∂b.

2. Apply differentiable binarization to the rendered silhouette image S. In our case, we convert a pixel value x ∈ [0, 1] in S to an approximately binary value as $11+e−100(x−0.5)≃{0,1}$⁠.

3. Zero out the pixel values that are greater than the threshold 0.5 to deal with the noise near the wrist caused by PyTorch3D differentiable rendering. The result of Steps 2 and 3 is a clean and nearly binarized silhouette $S¯$⁠.

4. Apply Laplacian operator to $S¯$ to get $ΔS¯$⁠. The contour $∂S¯$ is then obtained by $∂S¯=tanh(max{ΔS¯,0})$⁠, which maps the values on the contour to near 1 and all the rest to 0.

5. Do element-wise product between the resulting contour $∂S¯$ and the distance field to retrieve the corresponding distance values and then take the sum to get ContourLoss(S, B).

All of the aforementioned computations are differentiable. Therefore, we can readily include the silhouette loss in our method and end-to-end train the network.

We employ the mesh loss to measure the error in predicting the 3D vertices of the hand mesh. Like the pose loss, the mesh loss is also evaluated on both the unaligned and aligned predictions:

The unaligned vertex loss ℒ_V consists of the L1 losses for the predictions from the preliminary and refined meshes:

where V ∈ ℝ^M×3 is the ground-truth 3D vertices of the hand mesh, $V^$ is the preliminarily reconstructed mesh, and $V^ref$ is the final prediction of vertices after refinement.

For the aligned vertex loss ℒ_alignV, the predicted vertices are aligned with the ground truth using orthogonal Procrustes analysis as described in the aligned pose loss (4), and then the L1 loss is applied to the aligned and the ground-truth mesh as

Since the problem formulation of the task we aim to address differs from those commonly adopted in prior work, we have not found state-of-the-art methods that take exactly the same input setting and could, therefore, be considered for direct comparison with our approach. Note that the work of Lee et al. [15] is only posted on arXiv as a preprint without available code. Therefore, we do not include their method in the following comparisons. Moreover, their method requires depth maps during training, which is more restrictive than ours. In our experiments, we first compare our method with the state-of-the-art depth-based model A2J [34] on a synthetic dataset and show that transferring our task for an existing model to solve is probably not as straightforward as it may seem. Further, we use the real data in FreiHAND to compare our method with the three state-of-the-art RGB-based methods: I2L-MeshNet [23], MeshTransformer [19], and CMR (ResNet18) [5]. The results show that our method performs particularly well under the unaligned evaluation setting of 3D hand pose estimation, which is more practical for real applications.

We conduct experiments on the following two datasets.

Synthetic Dataset. This dataset contains 20,000 training and 2,000 test binary images synthesized using MANO [25, 28]. For each sample in the dataset, its hand-pose principal components θ ∈ ℝ⁶ are randomly sampled from the uniform distribution between [−2.0, 2.0), and each rotation angle of its 3D global orientation ℝ is randomly sampled from the uniform distribution [−π, π]. We use MANO to generate the corresponding hand shapes from the sampled parameters. The ground-truth 3D coordinates of the 21 hand joints can thus be automatically derived from the MANO-rendered hand shapes.

FreiHAND [37]. It is a common real-world RGB dataset with 32,560 training samples and 3,960 test samples. We binarize the segmentation masks available in the training set and use them as input for our model. However, since its original evaluation set does not provide segmentation masks, we resort to splitting its original training set into a random partition of 26,000 training, 3,280 validation, and 3,280 test images to achieve a reasonably fair evaluation of our method.

For quantitative evaluation, we report the standard metrics used by the FreiHAND dataset on the predictions of 3D joints and meshes:

• MPJPE: the mean per joint position error, which measures the Euclidean distance (in cm) between the ground-truth joints and the predicted joints.

• MPVPE: the mean per-vertex position error, which measures the Euclidean distance (in cm) between the ground-truth vertices and the predicted vertices of a mesh.

• AUC of PCK: the area under the curve of the percentage of correct keypoints, which is plotted using 100 equally-spaced thresholds between 0 cm to 5 cm.

• AUC of PCV: the area under the curve of the percentage of correct vertices.

• PA-MPJPE or PA-MPVPE: It first applies Procrustes alignment between the ground truth and the prediction and then calculates MPJPE or MPVPE. This metric aims to measure reconstruction error that neglects the effect of global rotation, translation, and scaling.

• mIoU and Dice coefficient: In addition to the preceding metrics used by the FreiHAND dataset for the evaluation of joint and mesh predictions, we also use the mean intersection over union and the Dice coefficient to evaluate the similarity between the input binary mask and the mask projected from the predicted hand mesh.

To demonstrate that state-of-the-art depth-based models cannot be easily transferred to the task with binary input, we compare our model with A2J [34] on the synthetic dataset. The input image of our model is binarized to {0, 255} while that of A2J is binarized to 0 (as the background) and the mean depth of joints to treat the depth-based model fairly. This way, even though the input image is binary-valued, we still provide sufficient statistics of depth information to A2J. As shown in Table 2, for the unaligned setting, our model achieves an MPJPE of 0.87 cm, and A2J reaches an MPJPE of 7.73 cm. The results indicate that A2J fails to learn to localize 3D hand joints accurately when the input is binary, even if the reference mean depth is given. In contrast, our method can recover the positions of the joints in the camera space. In terms of MPJPE after Procrustes alignment, our approach also outperforms A2J by a large margin, as presented in Table 2. Some other depth-based models are not quantitatively compared with ours for the following reasons. V2V-PoseNet [22] takes a 3D voxelized depth map as its input, which is expected to fail for silhouette-based input since the voxelization process cannot work for binary images. Hand PointNet [11] converts a depth map into a 3D point cloud and then uses it as the model’s input. Likewise, the conversion procedure cannot function properly for silhouettes, making it inadequate for the task of binary input.

For the comparison with state-of-the-art RGB-based methods, we select the leading models from the FreiHAND competition leaderboard at CodaLab,² including CMR [5] and MeshTransformer [19]. To make a fair comparison with our method, we use the officially released code to re-train their models on the FreiHAND dataset randomly partitioned by us, as mentioned in Section 4.1, under the settings of both binary and RGB input. The former is for performance observation, while the latter is primarily for convincing proof that these models are not trained imperfectly. As presented in Table 3, CMR achieves MPJPE and MPVPE of 4.31 cm with silhouette-based input, while MeshTransformer fails to localize 3D hand joints and mesh in the camera space. Our model reaches MPJPE of 3.56 cm and MPVPE of 3.57 cm, which outperforms the previous two methods by a large margin and is even comparable to the result of CMR with RGB input. The superior performance of our model in the unaligned case indicates that it has a better ability to reconstruct 3D hand mesh in the camera space, which is more practical in real applications. In terms of mIoU and Dice coefficient, our model achieves results similar to those of CMR.

When the predicted joints and vertices are aligned with the ground truth by Procrustes analysis, our approach reaches PA-MPJPE of 0.68 cm and PA-MPVPE of 0.69 cm, as shown in Table 4. These experimental results are significantly better than CMR’s performance under the setting of binary input and are comparable to CMR’s results with RGB input. Furthermore, unlike MeshTransformer, which totally fails in the unaligned case, our method performs stably well in both cases. Such a balanced behavior is preferable in our task since localization and detail recovery are vital for projecting a silhouette similar to the input one.

We show qualitative results on the test data. Note that the pictures illustrated in the main paper and this section demonstrate the results of the unaligned case. That is, we do not perform Procrustes alignment for visualization.

In Figure 3 and Figure 4, we show two columns of 15 sets of results side by side. Each set contains five images: the first image presents the input of the 2D binary mask. The second image shows the rendered silhouette using the predicted hand shape. The third image shows the predicted 2D skeleton projected onto the image space. Note that the color images shown here are merely for visualization; our method does not require any RGB data. The last two images depict the predicted mesh and 3D skeleton in the camera space.

As relatively clean masks are used for experiments in the main paper, we aim to explore the robustness of each model under the noisy silhouette setting in this section. To achieve this goal, we first train a Deeplabv3 with ResNet-50 backbone as the hand segmentation model that takes an RGB image of a hand as input and produces its estimated mask. Then, we utilize the estimated masks (i.e. noisy binary silhouettes) as inputs to train and evaluate both CMR and our method. Note that the silhouette loss is also calculated using the estimated masks in this experiment. The Deeplabv3 model reaches mIoU of 0.956 on the training data and 0.936 on the test data after being trained for seven epochs with a learning rate of 10⁻⁴. Despite the high mIoUs of Deeplabv3 predictions, by manually inspecting the produced silhouettes (see Figure 5 and Figure 6), we discover that finger poses in them are not reconstructed well, which can be considered a strong enough noise injected into the input of target models. As shown in Table 5 and Table 6, our method still outperforms CMR by a large margin in all evaluation metrics. Further, the performance degradation of our model is significantly less than that of CMR in both unaligned and aligned cases, demonstrating that our approach is more robust to noisy input and, thus, more suitable for real-world applications.

Our encoder’s backbone is based on ResNet18 [13]. PyTorch3D’s implementation [24] is employed for our differentiable renderer. We use the Adam optimizer to train our network with a batch size of 32. The initial learning rate is set to 10⁻⁴, and we use the ReduceOnPlateau scheduler to adjust the learning rate. The whole network is trained for 150 epochs, and the checkpoint with the lowest validation loss is selected for our final usage. For the FreiHAND dataset, we apply data augmentation of random rotation between [−π, π) and scaling between [0.9, 1.1) in the 2D image space. For the synthetic dataset, we randomly sample only one of the five views for each training data at every epoch to perform data augmentation. We set λ_J = 2 × 10⁻³, λ_alignJ =2 × 10⁻², λ_contour = 10⁻⁴, λ_V = 0.1, and λ_alignV = 1 to balance each loss term. λ_bce is set to 0 for our final model and 0.5 for the ablation study.

We evaluate different configurations of our method to analyze the effectiveness of each component and examine the effects of the loss functions.

In our ablation study, we use six pose PCs and set β to the mean shape, i e., β = 0. By comparing the first row of Table 7 and the penultimate row of Tables 3 and 4, we can observe that regressing shape parameters in the first stage of our model helps reduce the burden of refinement and achieve better performance in the aligned case.

As shown in Table 7, we explore the evaluation results on the FreiHAND dataset with the exclusion of different components. Our base model consists of Encoder, MANO Layer, and Differentiable Renderer. After incorporating the RefineNet module into our network with pose loss only, we observe that both MPJPE and PA-MPJPE decrease while MPVPE and PA-MPVPE undesirably increase. This result aligns with our expectation since the pose loss cannot impose strong enough constraints on the deformation of the preliminarily constructed hand mesh. Based on the observation, we further apply the mesh loss to the output of the RefineNet, which helps strengthen the refinement functionality and improve all evaluation metrics significantly.

To reveal the effects of other losses, we exclude them one at a time from our entire network. First, we remove the contour loss between the input binary image and the projected silhouette. The evaluation result shows that the model’s performance on both hand pose and shape estimation degrades, especially the unaligned ones, demonstrating the effectiveness of the contour loss for achieving more accurate global mesh registration. Second, we eliminate the aligned losses of both joints and vertices. Our experimental result shows that MPJPE, MPVPE, PA-MPJPE, and PA-MPVPE increase by 0.28, 0.35, 0.58, and 0.77, respectively. Such a significant deterioration in performance indicates that the aligned losses not only play crucial roles in the reconstruction of fine-grained gestures but also help to improve mesh recovery in the camera space. Finally, we remove all losses inserted before RefineNet. The result shown in the sixth row of Table 7 suggests that adding the pose loss and contour loss before the refinement stage can help our model improve the ability to localize the 3D hand mesh in the camera space.

We have also done some experiments to see whether adding the binary cross-entropy loss between the input binary mask and the projected silhouette has any positive effect. As listed in Table 7, directly incorporating the BCE loss into our entire architecture contributes nothing to the final performance. To understand the root cause of this counterintuitive phenomenon, we take away the RefineNet module and the aligned loss one at a time with the presence of BCE loss. The experimental results show that the BCE loss can enhance our model’s performance by a large margin when the aligned losses are absent. This implies that the BCE loss does not have much effect when the model learns the shape estimation task to a certain degree guided by the stronger training objectives, i.e., the aligned losses of joints and vertices.

When there is apparent ambiguity in the input silhouette, our method might predicts a distinct gesture that gives a very similar projected shadow. For instance, in the ground-truth image of Figure 7, the middle finger is extended, and the index finger is crooked. Our model predicts inversely, but the resulting silhouette is similar to the ground truth. This issue is quite challenging since the information of gradients in texture is lost in binary images.

We further explain why it is impossible to compare our method against Silhouette-Net on the HIM2017 dataset. HIM2017’s depth images include human bodies, and the captured hands are relatively small. The preprocessing step of this dataset requires cropping the hand segments by the given bounding boxes and resizing, which may lead to an unknown transformation between the camera space and the image space. The given camera intrinsics are thus inconsistent and misleading. Such an issue may be one of the reasons that depth supervision or multi-view is essential for a model (including Silhouette-Net) to succeed in HIM2017. Therefore, we consider HIM2017 inadequate for our task with single-view binary information.

This paper introduces the Mask2Hand network that learns to predict the 3D hand pose and shape from a 2D binary mask without relying on any RGB or depth information. Despite our task’s more challenging input setting compared with prior work on hand pose and shape estimation, we show that the proposed method can achieve comparable performance as state-of-the-art RGB-based and depth-based methods. With the parametric hand model and the differentiable rendering technique, we integrate the self-supervised mechanism into our end-to-end training process without human annotation. We propose several loss functions to model different aspects of 3D hand pose and shape estimation. Since our method explicitly learns to predict the global transformation of the hand, its better performance in the unaligned setting is thus promising for real applications.

This work was supported in part by the Industrial Technology Research Institute, Taiwan. We are grateful to the National Center for High-performance Computing for providing computational resources and facilities.