JointFormer: Joint-Enhanced 3D Human Point Cloud Completion Based on Transformer

Driven by the rapid development of 3D sensors, point clouds are emerging as an efficient data format for representing objects. Nonetheless, real-world point clouds often suffer from incompleteness and sparsity due to occlusion and sensor limitations. Therefore, point cloud completion has garnered widespread attention in computer vision and graphics, aiming to infer missing parts and densify sparse point clouds. This process benefits downstream tasks like human body reconstruction, object recognition, and pose estimation [11, 11, 9].

Compared to typical point cloud completion tasks, human point cloud completion poses more intricate challenges due to pose variability and complicated geometric relationships. The diverse poses of the human body demand highly adaptable algorithms to achieve coherent and realistic reconstruction across different postures. Moreover, the geometric relationships in human body point clouds, such as symmetry, are more nuanced than those in rigid objects, complicating the task of precise detail preservation and completion. In addition, the emergence of adversarial attack methods [3] for point cloud completion models has raised expectations for the robustness and resistance to interference of completion methods, which will enhance the security of downstream tasks such as 3D recognition and segmentation.

In recent years, deep learning-based point cloud completion has been flourishing. Some pioneering works on point cloud completion [20, 20, 31] directly handle 3D point cloud coordinates with an encoder-decoder architecture to generate complete point clouds. To avoid the information loss caused by max pooling operations in this architecture, some methods [29, 29, 34] incorporate Transformers to obtain detailed global features. However, when tackling non-rigid human point clouds with a body part such as a leg or arm severely missing, the global features extracted by existing methods are insufficient to infer the complete body parts.

Besides global features, another issue stems from the design of attention mechanisms for point clouds. Attention mechanisms have proven effective for point cloud analysis [33, 25]. Consequently, many recent methods have applied Transformers to learn structural features and long-range relationships within local regions of point clouds [34, 34, 22]. In these methods, the relationships between 3D points are usually explicitly introduced through position encoding.

However, spatial relationships introduced solely by position encoding seem insufficient. This affects the establishment of semantic relationships between points within local patches of the point cloud.

To address the aforementioned issues, we propose a novel human point cloud completion network called JointFormer. Our encoder is designed with a joint-enhanced strategy and consists of two branches: (i) encoding incomplete input to obtain local features and global features, (ii) encoding joints predicted from incomplete input to derive prior guidance for the complete point cloud. Secondly, we design a channel-spatial attention Transformer (CSAT) to incorporate positional information of features into the computation of point attention scores, extending the original channel attention and capturing more comprehensive local point cloud semantic information. Our main contributions can be summarized as follows:

1. We design a dual-branch joint-enhanced encoder. It predicts joints and further yields refined global features and local features, which introduces human pose prior structural information for the subsequent decoding phase.

2. We devise a channel-spatial attention Transformer (CSAT), which can better extract regional point cloud semantic relationships during both encoder and decoder in a simple yet effective way.

3. We evaluate our network on two self-crafted human point cloud datasets and public dataset PCN, which demonstrates our method achieved excellent performance.

Early works [13, 13, 19] convert irregular point clouds into 3D voxels and then perform point cloud completion based on 3D convolutions. However, this conversion requires significant computational resources and inevitably leads to geometric information loss. Consequently, in recent years, approaches that directly operate on point clouds have become mainstream. With the success of Transformer in Natural Language Processing (NLP), there has also been a surge of research exploring the potential of Transformer in point clouds. Therefore, we categorize related works into two groups: Non-Transformer-based methods and Transformer-based methods.

PointNet [16] and its successor, PointNet++ [17] have pioneered the application of deep learning directly on point clouds. In point cloud completion, PCN [31] firstly proposes using an encoder-decoder architecture, demonstrating that point-based completion methods have higher generalization performance and robustness compared to voxel-based methods. Pf-net [4] combines GAN with a hierarchical decoding strategy to predict missing parts. Moreover, Pmp-net [21] simulates the movement of points during completion and constrains the total distance of points to obtain a point-wise unique motion path. HyperCD [8] proposes measuring point cloud distances in hyperbolic space instead of Euclidean space, which can alleviate the vulnerability of CD to outliers. Flattening-Net [32] converts irregular 3D point clouds into regular 2D point geometry images, serving as a versatile representation for reconstruction and other tasks, and providing a novel perspective on point cloud completion.

PoinTr [29] explicitly models the local geometric relationships of point clouds with Transformer, better learning and preserving structural information of point cloud. SnowflakeNet [26] leverages skip-transformer mechanism to infer the splitting patterns of the current layer from those of preceding layers within the SPD process. Seedformer [34] introduces patch seeds and extends Transformer to point generation operation, completing point cloud in a coarse-to-fine manner. Pmp-net++ [22] enhances the learning of point features with Transformer. Cross-PCC [24] explores the unsupervised method assisted by single-view images, and also leverages Transformer to capturing point relationships in 3D feature extraction, which provides valuable insights for point cloud completion without a large labeled dataset. ProxyFormer [6] introduces point proxy representation and design a missing part-sensitive Transformer, enabling the network to better predict missing parts. However, when applied to human point clouds with severe deficiencies, the performance of these methods seems mediocre due to the lack of internal structural relationship exploration.

In recent years, there has been a growing body of research on human point clouds. Some methods [35, 7, 23, 1, 1] propose modeling the complex surface structure directly from point clouds and then achieving an accurate action recognition and pose estimation of the human body. There are also methods dedicated to addressing the challenges of human reconstruction starting from point clouds. The previous method [5] extracts and maps skeleton features with PointNet++ [17], then regresses to obtain SMPL [10] parameters for human shape reconstruction. VoteHMR [9] is proposed to recover human mesh from point clouds, which can effectively encode human body geometric information and has strong robustness to noisy inputs with self-occlusion and missing areas. It is evident that human point clouds play a crucial role in pose estimation and reconstruction. Human point clouds are clearly vital for pose estimation and reconstruction. Consequently, our research aims to address the issue of incomplete human point clouds to improve performance in these downstream tasks mentioned above.

The overall architecture of Joint Former is illustrated in Figure 1, which consists of three parts: joint-enhanced encoder, generator, and decoder.

Our joint-enhanced encoder consists of a surface point encoding branch and a joint encoding branch. Given an incomplete input P, the former branch extracts global features ℱ_ɡ, local features ℱ_p, and corresponding patch center P_p. In the joint encoding branch, with generated point-wise votes {s_i,o_i,f_i}, where i represents the i-th point, aggregating those with the same semantic scores s_i, the human joints J are produced. Afterward, clustering partial input in terms of obtained joints, an ordered point patch sequence is obtained and fed into the Transformer block to extract joint features ℱ_joints. Concatenating ℱ _joints with the global features ℱ_g mentioned above, we obtain fine-grained global features $ℱg*∈ℝ2×Ng×1$⁠.

In generator, we feed Ƥ_p, ℱ_p and $ℱg*$ into UpTrans-CSAT, while CSAT is applied to better capture the relationships among features. After a few MLPs, a rough yet complete seed point Ƥ_sd can be generated.

The decoder is devoted to recovering the complete point cloud reliably with fine details. We combine the upsampling process [34] and CSAT, named UpLayer-CSAT. Specifically, each layer interpolates seed points with seed features to obtain upsampled point cloud through CSAT. During all three layers, a finer point cloud P_i(i = 1, 2, 3) is generated while the previous output serves as its input. Notably, to better preserve the detail in input and fully utilize predicted joint information, the entire decoder takes P₀ ∈ ℝ^No ׳ as input, which is obtained by concatenating P, J, and P_sd.

For input incomplete human point cloud P ∈ ℝ^N×3, the joint prediction module generates human joints $J∈ℝNk×3$⁠, adhering to SMPL (Bogo et al., 2016), where N_k is 24.

Specifically, as shown in Figure 2(a), we employ PointNet++ [17] as the backbone to extract joint feature f_joint.

We leverage voting module [15, 9] to generate point-wise votes {s_i,o_i,f_i} due to its success on predicting offset coordinates and features for each seed point, which will benefit the joints discovery even with severe noises, missed areas, and occlusions.

The module consists of MLPs, followed by independent heads for body part segmentation, joint regression, and feature updating respectively. s_i is formed by the action of an FC layer and softmax. The joint regression head adopts an FC layer to output offsets o_i ∈ ℝ³ for each point p_i, and also aggregates more reasonable contextual information to define joint positions.

The feature updating head uses residual connections to update vote features fi = f_jointi + Δf_i, i = 1,,N. Δf_i is extracted from a shared MLP with an FC layer.

Once we have obtained the point-wise votes, we can proceed to the group and aggregate them to obtain joints. Specifically, the coordinates j_k of each joint can be represented as:

Where s_i represents semantic content indicating the body part, p_i represents each point of the partial input.

Throughout joint estimation, the point-wise labels y_i can provide auxiliary support. Practically, we set all labels to a specific value. This ensures that the generated points align with the input human point cloud in the same coordinate system while also promoting model convergence. Moreover, the predicted joint coordinates are regularized against the ground truth with L2 loss during training.

As shown in Figure 2(b), given input incomplete point cloud P ∈ ℝ^N×ɜ and predicted joint $J∈ℝNk×3$⁠, firstly, we partition input incomplete point cloud with KNN to obtain the neighbors of each joint $Jneigh∈ℝNk×K×3$⁠. Then, we compute the distances among joints, followed by sorting to determine the order of joints. Finally, iterating through all 24 joints, the index tensor J_idx labeled with joint order information is generated, while also sorting $Jneigh∈ℝNk×K×3$ accordingly.

Having obtained the ordered joint coordinates J and the ordered joint neighbors J_neigh, we first encode J_neigh using PointNet [16] to derive the features of joint neighbors. Then, we perform attention calculation on this ordered sequence of point features with a dual masking strategy, which masks some previous tokens of the current token. It can be represented as follows:

where Q, K, and V are obtained by encoding T with different weights along the channel dimension D. The masked positions M^d are set to 0 if masked, and 1 otherwise.

Eventually, our joint encoder is composed entirely of transformer decoder blocks with a dual masking strategy, obtaining joint feature ℱ-joints.

CSAT complements the point attention mechanism by excavating relationships within attention weights in both channel and spatial dimensions.

Given seed points $Psd∈ℝNsd×3$ and corresponding seed features $ℱsd∈ℝNsd×D$⁠, the query vector Q are generated by concatenating them and passing through a MLP. The key vector K in UpLayer-CSAT is composed of output features from the previous layer to retain features from input. Then, the value vector V are composed of concatenated Q and K, as shown in Figure 3(a). CSAT calculates channel-wise attention weights $a^ij$ after subtracting each point from its k nearest neighbors 𝒩(i) as follows:

where α, β, and γ are feature mapping functions MLP and linear layers. $Q,K∈ℝDh×Nl×K$ represents fused features of joint-wise and point-wise features.

Then, as shown in Figure 3(b), two pooling operations is utilized to generate average-pooled features ℱ_avg ∈ ℝ^1×H×W and max-pooled features ℱ_max ∈ ℝ^1×H×W across the channel in $a^ij$⁠, which are then concatenated and fed into a convolution layer. We compute point attention weights $a^ij*$ as follows:

where σ denote sigmoid function and f^7×7 represents a convolution operation with the filter size of 7 × 7.

Through elemental-wise multiplication of computed attention weights α_ij and value vector, we obtain the features h_i of new points around each seed point.

where ψ is a feature mapping function. exp() denotes softmax function used to normalize the computed weights $a^ij.V∈ℝDh×Nl×K$ represents point-wise features.

The aggregated features of the K-nearest neighbors of all seed points generate the upsampled features, doubling as the key vectors for the subsequent iteration in the skip connection. Finally, a set of point offsets is obtained using a shared multi-layer perceptron based on the generated new point features. Adding these offsets to the replicated original points results in the generation of new upsampling points.

We propose a combined loss of Chamfer Distance (CD) and joint L2 loss in our end-to-end human point cloud completion network, defined as:

where λ_cd, λ_joint represent the weights of hierarchical CD loss and joint estimation loss respectively. In this paper, we set λ_cd to 1 and λ_joint to 0.5.

Specifically, generated point clouds at multiple stages of point cloud completion process are supervised, denoted as ℙ ∈ {Ƥ_seed, Ƥ₁, Ƥ₂, Ƥ₃}, referred to as hierarchical CD loss ℒ_CD. Ground truth $Pgt∈ℝNt×3$ is aligned with the generated point cloud in resolution with Farthest Point Sampling (FPS). The ℒ_CD can be expressed as:

Additionally, to better utilize joint features obtained based on joint estimation, we incorporate joint loss into the network to supervise the output of the joint estimation module. The joint estimation loss ℒ_Joint is formulated as:

We utilize widely adopted metrics including Chamfer Distance (CD), Chamfer Distance with L1 norm (CD-ℓ1), and F-score.

CD evaluates the overall distribution, which ensures generated point cloud represents the surface of the target. To alleviate the sensitivity of the L2 norm to individual outliers, CD-ll is adopted as a more balanced metric. Additionally, the F-Score assesses the similarity between the generated and the the target.

In the joint-enhanced encoder module, we set the point-wise labels y_i to be 0.001, the value of k in KNN to be 32, and the groups of KNN N_k to be 24. We conducted extensive experiments on several NVIDIA RTX4090 GPUs. We trained our JointFormer end-to-end on PyTorch for 400 epochs. We used the Adam optimizer with β₁ = 0.9 and β₂ = 0.999.

Given the absence of a publicly available dataset of partial human body point clouds, we referred to [14] to create two high-quality multi-view incomplete human body point cloud datasets: THuman2.0 [28] and AMASS [12].

THuman2.0: To simulate the scenarios of incompleteness and sparsity that occur in the real world, we first rendered 10 depth images from different random viewpoints based on the model. Subsequently, transforming the generated depth images into the world coordinate system, we obtain partial point cloud data. Figure 4 shows an example of point cloud data obtained with a focal length of 100.

Finally, the THuman2.0 dataset consists of 4200 training data, 530 validation data, and 530 test data. Each ground truth point cloud is generated by evenly sampling 16384 points from the model surface. During experiments, the batch size is 8, and the initial learning rate is 0.001 and decays by 0.1 every 100 epochs.

AMASS-part: Considering the limited variety of human models and poses in the THuman2.0 dataset, we further developed a new, more diverse dataset from part of AMASS [12], named AMASS-part. AMASS encompasses a rich collection of human motion sequences stored as SMPL parameters. Therefore, we first convert the SMPL parameters into 3D human models and then generate point cloud data using the same approach as in THuman2.0 mentioned above.

Finally, AMASS-part consists of 4489 human models. After sampling from 10 random viewpoints, the training set contains 43,830 samples, while the test set and the validation set each have 5,530 samples. We raised the batch size to 48; the initial learning rate is 0.0005.

We compared our JointFormer with several baseline methods on our THu-man2.0 dataset. For fairness, we retrained these methods with their optimal parameters respectively.

point clouds sampled from 10 random viewpoints. The resolution of ground truth is 16384.

The joints are extracted from SMPL parameters.

As shown in Table 1, we report Chamfer Distance, Chamfer Distance with L1 norm, and F-Score with several previous methods. Generally, lower Chamfer Distance indicates more accurate reconstructive shape. Compared to the second-ranked AdaPoinTr, JointFormer reduces CD-ℓl by 0.48, which is 8.6% lower, while CD is 10.0% lower and F-Score is 4.5% higher. As indicated by results, the global guidance introduced through joint information in human poses and the exploration of spatial relationships indeed assist in the completion of incomplete human point clouds. Our JointFormer, through its novel designs, surpasses them by a significant margin.

As shown in Figure 5, we can visually compare with several previous methods. The 1st column displays incomplete point clouds with varying degrees of missing data. The 1st point cloud lost data on the back and left leg due to self-occlusion. The 2nd and 3rd point clouds miss back and front respectively. The 4th point cloud suffers significant data loss in the body and legs. Each row shows the results of different methods for completing the point clouds for each partial data. In the 1st and 2nd rows, our method has finer details on the right hand and neck of the human body, with a more even distribution of surface and fewer outliers. Especially, it can be observed that our Joint-Former has the smoothest surface and provides a more detailed and credible completion of the left leg in the 1st row. In the 3rd row, facing the severely missing on the neck of the human body, our method has a more reasonable edge, while Seedformer and AdaPoinTr introduce some outliers at the toes and uneven distribution of the right arm. In the 4th row, where the legs are almost entirely missing, JointFormer performs better at detecting the presence of the right foot compared to other methods. Meanwhile, SnowflakeNet and AdaPoinTr, despite having fewer outliers, prematurely halt the completion process. In conclusion, our JointFormer has a stronger ability to restore details of human body parts and smooth surface points.

As shown in Table 2, we conduct experiments under different difficulty levels to further evaluate the performance of JointFormer. CD1-S, CD1-M, and CD1-H represent CD-ℓ1 when the viewpoint focal length is set to 100, 150, and 200. Considering that training data was generated at a focal length of 100, as the focal length increases, the discrepancy between the input and training data grows. Consequently, the difficulty of model inference increases. However, compared to previous methods, our model performs well at all levels and averages, except for slightly higher than PoinTr in hard level (focal length is 200). Compared to the second-ranked Seedformer, JointFormer decreased by 8.2% on CD-ℓ1 Avg. This indicates that our method exhibits superior generalization performance when applied to data not encountered during training.

On the AMASS-part dataset, we also conduct experiments for our method and other five state-of-the-art methods. Compared to THuman2.0, the AMASS-part is sampled from human motion sequences and includes more complicated human postures, making the completion task significantly more challenging.

The results are shown in Table 3. We see our method also achieves the best performance in this more challenging dataset. Compared with Seedformer, not only the CD-ll performance drop of our method is up to 2.8, but also the CD and F1 performance much better. We also visualize the results in Figure 6 to show the excellent performance of our method on human motion sequences. Compared with Seedformer, in the 1-st row, our model delivers a more complete point cloud of the human head. In the 2-nd row, it provides a more accurate and reliable point cloud of the hands. In the 3-rd row, it produces a finer and more detailed point cloud of the left side of the body.

The evaluation results of the joint-enhanced encoder are shown in Table 4. The baseline model A is the basic point transformer for point cloud completion, which uses encoder-decoder architecture with point Transformer. In this model, we extract local and global features with SA layers, while the decoder consists of point transformer without spatial attention. We then add the joint encoding branch in the encoder (model B). We see the joint encoding branch improves the baseline by 0.13 in Chamfer Distance with L1-norm. Furthermore, we incorporate the joint regularization loss (model C) to investigate its impact on the completion results. It can be seen the addition of regulariza-tion loss slightly improves the completion performance. That is, the human joints predicted by a joint encoding branch can indeed provide reliable global information for guiding completion tasks, particularly when the input point cloud has substantial gaps or severe missing regions.

To verify the effectiveness of our CSAT (Channel-Spatial Attention Transformer), we conducted ablation experiments as shown in Table 4. Model E represents our JointFormer. For comparison, we create model D by removing the joint encoding branch from the encoder. Notably, even without the joint encoding branch, model D still integrates CSAT into the surface point encoding branch, as well as into the upsampling Transformer blocks within the generator and decoder. This leads to a decrease in the CD-ll score to 5.14 compared to model A (lower is better). The results confirm the effectiveness of our joint-enhanced encoder.

Additionally, we evaluated CSAT on the public dataset PCN. As shown in Table 5, our model performs better than previous methods in all 8 categories except for lamp. This indicates that the introduction of additional spatial attention does not bring good improvement effects for objects with the shape of a lamp. However, on average, the CSAT can better assist Transformers in capturing neighboring relationships in unordered and structurally irregular point clouds, which is crucial for point cloud completion tasks. Comparing Seedformer and Mode D, we can be seen that CSAT only introduces a slight improvement (0.03) on the PCN dataset, while there is a 13% improvement on the human point cloud dataset. This further validates the effectiveness of our CSAT.

In this paper, we propose JointFormer, a novel method for point cloud completion in non-rigid objects, particularly focusing on the human body. By fully leveraging self-attention mechanisms, our method effectively captures both local and long-range structural relationships among unordered points. In cases of severe missing situations, the joint encoding branch plays a crucial role in guiding the inference process. Extensive comparisons and ablation studies underscore the superiority of JointFormer, demonstrating its capability to outperform state-of-the-art methods. Additionally, we have introduced two new, and more challenging datasets specifically designed for human point cloud completion. Further exploration of our architecture in other 3D human reconstruction tasks could present an exciting direction for future research.