Shared multi-branch framework and enhanced feature fusion for few-shot semantic segmentation

Image segmentation technology has widespread applications in fields such as medical image analysis, autonomous driving, and remote sensing. Semantic segmentation, a critical task within image segmentation, assigns class labels to each pixel in an image. While supervised semantic segmentation has made significant progress, it requires extensive labeled data, incurring high manual annotation costs. Practical constraints like labor, budget, and time often hinder the acquisition of large, accurate datasets. These challenges have driven the development of few-shot segmentation (FSS), which aims to segment new categories with minimal labeled data.

Recent advancements in semantic segmentation have introduced strategies to address limited data and complex scenarios. Lv et al. (2024) improved performance by integrating RGB and thermal (T) data with a context-aware interaction module, although their method is restricted to RGB-T scenarios. Qin et al. (2024) proposed a pyramid fusion transformer for multi-scale fusion, but its computational cost limits applicability in few-shot settings. Beyond segmentation, small object detection and feature fusion have inspired further innovations. Wang et al. (2024a) demonstrated that feature-level augmentation significantly enhances segmentation accuracy, while Wang et al. (2024b) showcased the effectiveness of lightweight architectures for handling complex data scenarios. In addition, Lin et al. (2024) further enhanced real-time segmentation accuracy by distilling rich knowledge from pre-trained vision-language models into lightweight segmentation frameworks, proving beneficial in resource-constrained environments.

Chen et al. (2024a) introduced a feature attention mechanism to enhance robustness against adversarial scenarios. In the specific context of FSS, Yu et al. (2023) improved feature distinction through background clustering, effectively reducing annotation needs. Moreover, recent research has shown promising directions in using lightweight, attention-driven architectures for segmentation tasks with minimal computational cost. For instance, Li et al. (2024) proposed a lightweight plank grading method using a multichannel spatial attention mechanism, which successfully balanced segmentation accuracy and computational efficiency. Do et al. (2025) developed a two-stage pipeline integrating semantic and instance learning for automated cell segmentation, demonstrating significant improvements in segmenting fine-grained biological structures with limited annotations.

Despite the emergence of powerful self-supervised foundation models like the DINO series (Oquab et al., 2023), which provide exceptional geometric structural priors, or large-scale vision-language models like Contrastive Language-Image Pre-training (CLIP), a significant challenge remains in effectively harmonizing these heterogeneous features for precise dense prediction in data-scarce environments. While foundation models excel in general representation, they often struggle to balance high-level semantic alignment with the fine-grained local textures required for complex segmentation tasks.

This study addresses these identified gaps by proposing a custom-optimized multi-branch backbone that strategically integrates heterogeneous models for superior local and global feature extraction. To mitigate background interference in the self-support setting, adaptive weighted masks are utilized to dynamically highlight discriminative foreground regions. Furthermore, an enhanced multi-scale fusion technique is developed to bridge the gap between high-level semantics and low-level structural details, enhancing the model’s representational richness while maintaining computational efficiency. Table 1 compares the key features of our method with prior works, including SSP (Fan et al., 2022), DFESN (Zhou, 2023), PANet (Catalano et al., 2024) and RestNet (Huang et al., 2023). Our proposed method uniquely synergizes a multi-branch architecture, shared networks, and weighted masks to offer a comprehensive and robust solution to few-shot semantic segmentation challenges.

In summary, this study addresses the inherent challenges of feature loss and noise interference in few-shot scenarios through the following contributions:

• Synergistic multi-branch backbone: A custom-optimized architecture integrating ResNet, EfficientNet and CLIP to bridge the gap between fine-grained local structures and global semantic priors.

• Adaptive weighted masks: A dynamic module that prioritizes discriminative foreground regions, mitigating background noise more effectively than traditional binary masking.

• Enhanced multi-scale feature fusion: A sophisticated fusion module that harmonizes high-level semantics with low-level details, optimizing the trade-off between representational richness and computational efficiency.

The rest of this paper is organized as follows: Section 2 reviews related works, Section 3 presents the proposed method, Section 4 details the experimental results, and Section 5 concludes the study.

This section reviews key methods that serve as references for our study in FSS. These include the self-support matching strategy, the multi-branch DFESN architecture, PANet’s ensemble techniques and RestNet’s use of residual connections. These approaches have advanced feature extraction and segmentation accuracy with limited labeled data, forming the foundation for our method.

SSP introduces a self-support matching framework, hypothesizing that parts of the same object exhibit higher similarity than those of different objects. Analysis on the PASCAL-5i dataset (Everingham et al., 2010) using ResNet-50 revealed that foreground pixels show greater similarity, while background pixels display partial similarity. Experiments comparing traditional support prototypes and self-support prototypes demonstrated SSP’s advantages in confidence, generalization, and efficiency. SSP outperformed existing methods across various settings, addressing intra-class variation and achieving strong results. However, SSP relies on pre-existing features, hindering fine-grained detail capture in complex scenarios with heterogeneous objects or intricate backgrounds. Performance also declines with domain shifts or limited support images.

DFESN is a dual-branch network combining ResNet for local features and Vision Transformer for global features. ResNet uses channel attention, while the Vision Transformer processes 16 × 16 patches with linear mapping and multi-head self-attention. On PASCAL-5i, DFESN achieved higher mean Intersection over Union (mIoU) and FB-IoU in one- and five-shot settings, demonstrating its advantages. However, DFESN’s dual-branch architecture incurs high computational costs, reducing real-time applicability. It also struggles with fine-grained feature fusion, especially with significant object scale variations or complex backgrounds.

Catalano et al. proposed feature volume fusion, integrating multiple backbone networks to capture richer details. Although this increases training parameters, FSS models generalize to unseen categories after a single training session, unlike traditional supervised models. PANet with feature volume fusion showed improved segmentation performance on PASCAL-5i and COCO-20i datasets, highlighting the potential of ensemble strategies.

RestNet improves cross-domain (CD) FSS by addressing semantic segmentation challenges in unseen domains with limited labeled samples. It introduces the intra-domain residual enhancement (IRE) module to preserve the original discriminative space via residual connections, enhancing support-query feature alignment and minimizing knowledge loss. The IRE module operates without additional parameters or target-domain training, maintaining model performance. Intra-domain matching is quantified by calculating support-query similarity and identifying pixel pairs with positive similarity values, termed intra-domain active matches. These matches reflect the model’s ability to retain intra-domain knowledge after CD projection. RestNet activates more intra-domain matches than existing methods, effectively leveraging support-query information. However, RestNet struggles when domain gaps between support and query images are significant, as residual enhancement may fail to fully align features, reducing segmentation accuracy. Its focus on intra-domain matches also limits its effectiveness in fine-grained segmentation, particularly for small objects or complex backgrounds.

The proposed architecture is illustrated in Figure 1. To enhance few-shot semantic segmentation, this paper introduces a shared multi-branch framework built upon the self-support matching strategy proposed in SSP. Figure 2 further depicts the feature extraction module incorporating three key components: a multi-branch shared network, weighted mask features and enhanced feature fusion.

Adding a multi-branch architecture helps improve the accuracy of few-shot semantic segmentation tasks for several reasons:

• Enhancing feature diversity: Different backbone models have their own strengths and characteristics in feature extraction. By combining these models, the resulting feature representations become richer and more diverse. This helps the model learn sufficient features even with limited samples, thus improving segmentation accuracy.

• Reducing over-fitting: The multi-branch architecture can integrate results from different feature extraction paths, reducing the risk of over-relying on any single path’s training data and thereby enhancing the model’s generalization ability.

• Strengthening feature representation: The multi-branch model can use ensemble learning methods to fuse features from different branches. This fusion captures critical information from the input data, providing sufficient clues for accurate semantic segmentation even in few-shot scenarios.

• Integrating multi-scale information: Different backbone models may excel at capturing features at different scales. For instance, some models might be better at capturing global information, while others excel at capturing fine details. The multi-branch architecture effectively integrates these multi-scale features, which can be considered as feature maps generated from a feature pyramid network.

• Enhancing model robustness: The multi-branch architecture incorporates redundancy to increase the model’s robustness against abnormal data or noise. When one branch fails to handle certain anomalous data correctly, other branches can still provide accurate feature representations, thereby improving the overall model’s robustness and generalization capabilities.

In our method, the multi-branch shared network comprises ResNet (He et al., 2016), EfficientNetV2 (Tan and Le, 2021) and the CLIP (Radford et al., 2021) visual encoder:

• ResNet: This deep residual network introduces a residual learning framework that effectively addresses the vanishing gradient problem in deep networks, enhancing both accuracy and training speed while maintaining depth. ResNet can learn rich, high-level feature representations, making it a widely adopted choice for computer vision and few-shot semantic segmentation tasks.

• EfficientNetV2: The second backbone network, EfficientNetV2, improves upon EfficientNet by further reducing computational resource requirements while increasing accuracy by 2%–3% points. This improvement is achieved through “compound scaling,” a method that simultaneously adjusts the network’s depth, width, and resolution to optimize performance and efficiency. EfficientNetV2 is designed to deliver high accuracy and speed across various application scenarios.

• CLIP visual encoder: The final backbone network uses the CLIP visual encoder due to its strong zero-shot learning capabilities. Zero-shot learning allows the model to be applied directly to new classification or segmentation tasks without specific task training, which is particularly valuable for few-shot semantic segmentation. CLIP’s zero-shot learning ability offers flexibility and efficiency, enabling accurate inference and decision-making based on pre-trained knowledge, even with minimal or no training samples. The pre-trained image encoder provides powerful feature representations, even in scenarios where large amounts of labeled data are not available.

Furthermore, ResNet and EfficientNetV2 have advantages in capturing local features due to their model design but lack the ability to consider global features. In contrast, CLIP, having learned a large number of image-text pairs during pre-training, can understand the overall structure of images well. This global feature extraction capability allows CLIP to capture the core content and contextual information of images, which is particularly beneficial for tasks requiring an understanding of the overall scene. Based on these advantages, incorporating the CLIP visual encoder as one of the backbones for FSS can enhance the model’s segmentation ability effectively.

In the experiments of our method, some neural network layers are frozen during model training. By freezing the initial layers, the general features learned from the pre-trained model are retained, avoiding the need to relearn these fundamental features for new tasks, significantly reducing the number of parameters to be updated and speeding up the training process. The deeper layers of the neural network, typically responsible for learning high-level features and making final decisions, are trained to adapt better to the specific task requirements, learning more targeted high-level features.

Here are detailed experimental descriptions of the three backbone branches. The first backbone network in this paper is ResNet-50, based on the ResNet architecture with a total of 50 parameter layers. The advantage of ResNet lies in its powerful ability to extract low-level features such as edges, textures, and colors. ResNet consists of five stages, and this study uses only the first three stages. According to MLC (Yang et al., 2021) experimental results, removing the last stage and the final nonlinear transformation layer (ReLU) of the backbone network can enhance the model’s generalization ability. In the proposed method, Stages 1 and 2 of ResNet are frozen, while Stage 3 is trained. The features extracted by ResNet are denoted as:

EfficientNetV2 optimizes the model’s depth, width, and resolution through the compound scaling strategy, achieving a balance between computational efficiency and model performance. To enhance low-level feature extraction without significantly increasing the number of parameters, EfficientNetV2 can provide efficient performance. This study freezes the first five blocks of EfficientNetV2, retaining the pre-trained weights of these layers to leverage their low-level feature extraction capabilities, ensuring efficient operation in few-shot scenarios. The features extracted by EfficientNetV2 are denoted as:

For the CLIP visual encoder, given its strong zero-shot capability, meaning it can perform well on new tasks without specific training, the weights are fully frozen, and no gradient updates are performed. The features extracted by the CLIP visual encoder are denoted as:

In Figure 3, weighted mask features involve multiplying the features generated by the backbone with the mask, which helps the model better capturing the relationships between features and enhance the perception of subtle features. By multiplying the features with the mask, the relevant parts of the features are strengthened, while the irrelevant parts are suppressed, thereby enhancing the perception of the target area. Subsequently, performing two rounds of average pooling can further weight the features, focusing more attention on information useful for the target while reducing the influence of unnecessary noise. This operation can improve the accuracy of the model in detecting target areas, making it more suitable for various complex semantic segmentation tasks. The calculation of the target area is defined as:

The weighted features are calculated as:

where $M$ represents the binary mask, which indicates the region of interest within the feature map $F1$⁠. The feature map $F1$ is extracted from the ResNet backbone, which specializes in capturing local features. This ensures that the binary mask $M$ effectively highlights the target regions within the image. $H$ and $W$ denote the height and width of the feature map, respectively. $sum$ represents the summation of all elements in the mask $M$⁠. The variable $ϵ$ is a small constant added to avoid division by zero, where $ϵ$ is set to 0.0005 in experiments.

Enhanced feature fusion references the feature fusion method adopted from ensembling networks, which has several significant advantages. This process is also illustrated in Figure 3. First, features extracted by different backbone networks have distinct characteristics and expressive capabilities. Integrating these features enhances the model’s ability to capture the diverse characteristics of the target, thereby improving the model’s perception and understanding of the target area. Different backbone networks may capture different information about the target at various levels and from different perspectives. By fusing this information, a more comprehensive and rich feature representation can be provided, further enhancing the model’s understanding of the target.

The steps in the proposed method involved using three different backbone networks to extract features from the input image. First, the features generated by ResNet (⁠$F1$⁠) are weighted and fused with the mask to improve the model’s accuracy and robustness in the target area, enabling better distinction between the foreground and background areas. Next, features extracted from EfficientNetV2 are denoted as $F2$⁠, and those from the CLIP visual encoder are denoted as $F3$⁠. $F2$ and $F3$ are first concatenated and then dimensionally reduced to match the dimensions of $F1$⁠. The dimensional reduction involves performing a 1 × 1 convolution operation on the concatenated features $F′2+3$⁠, followed by a nonlinear transformation and dropout with a probability(⁠$p$⁠) of 0.1, resulting in the feature fusion output $F2+3*$⁠, which are given by:

Finally, an IRE module is adopted as the residual connection to retain the necessary information in the original space. After concatenating F₂ and F₃, the dimensionality-reduced result F^*₂₊₃ is added to the previously weighted mask feature F^*₁. Simultaneously, the feature F₁ extracted by ResNet is connected through a residual connection. Finally, features are fused in a weighted manner to generate the final support feature F_s. By applying masked average pooling to F_s, the support prototype is generated according to:

Subsequently, $Ps$ is the result of applying masked average pooling on the feature map $Fs$ given by:

Unlike conventional multi-branch designs that primarily focus on model ensembling, our shared multi-branch backbone is intentionally constructed with heterogeneous feature priors. Specifically, ResNet and EfficientNetV2 are partially fine-tuned to emphasize local texture and low-level structural cues, while the CLIP visual encoder is fully frozen to provide global semantic representations learned from large-scale image–text pre-training. This asymmetric training and freezing strategy enables effective integration of local and global information under limited supervision, which is not explicitly addressed in standard multi-backbone frameworks. Subsequently, the proposed weighted mask feature is not used merely as an attention mechanism but is tightly coupled with the ResNet branch to enhance foreground-focused local feature aggregation. By combining mask-based weighting with successive average pooling, the method suppresses background interference while preserving discriminative local features critical for FSS, especially when support samples are scarce. Finally, the enhanced feature fusion module is designed to balance representational richness and computational efficiency. Instead of directly concatenating multi-scale features, features from EfficientNetV2 and CLIP are first fused and dimensionally aligned before being integrated with the weighted mask features via residual enhancement. This structured fusion strategy reduces redundancy while preserving complementary information across scales and domains.

This section presents the experiments conducted using the PASCAL-5i dataset to evaluate segmentation accuracy. First, we introduce the evaluation metrics used to assess segmentation performance. Then, we discuss and analyze the classification results. Finally, we compare the results to demonstrate the contributions of this experiment.

The development environment and specifications used are as follows:

• CPU: Intel(R) Xeon(R) Gold 5218 @ 2.30 GHz.

• Memory: DDR4 – 128 GB.

• GPU: NVIDIA RTX 3090 (1 unit).

• Operating system: Ubuntu 20.04 LTS.

• Programming language and libraries: Python 3.10.

These hardware and software configurations ensured efficient experimentation and processing of the segmentation tasks.

PASCAL-5i is a dataset used for FSS, containing a large number of images and corresponding labels. These images are derived from the PASCAL VOC 2012 dataset, which is a widely used open dataset for image classification, segmentation, and object detection. The labels in the PASCAL-5i dataset classify the objects appearing in the images, helping researchers train and evaluate the performance of computer vision models. This dataset is commonly used to measure the accuracy and generalization capabilities of object detection and classification algorithms. The PASCAL-5i dataset consists of a total of 5,000 images, categorized into 20 different classes. Each image is annotated with labels that indicate the objects present and their locations within the image. The dataset is divided into fourfolds, with each fold containing five classes. During training, the model learns from 15 classes by training on three of the folds, while the remaining fold, containing five unseen classes, is used for testing. This setup allows for evaluating the model’s performance on new, unseen categories, thus testing its generalization ability.

In the training process for few-shot semantic segmentation, the dataset is first divided into fourfolds. During each training cycle, three of these folds are used for training, while the remaining fold is reserved for testing. For example, in the first cycle, folds 1–3 are used as training samples, and fold 0 serves as the testing sample. This process is repeated four times, with the folds rotated so that each fold is used as the testing set once. This cross-validation approach ensures that the model is thoroughly evaluated and helps to reduce overfitting, leading to a more robust assessment of the model’s performance across the entire dataset.

After each test, the Intersection over Union (IoU) for that specific fold is calculated. Once all fourfolds have been tested, the mIoU across all folds is computed to evaluate the overall performance of the model. This approach ensures that the model is tested on unseen classes during each cycle, providing a robust measure of its generalization capability. By rotating the folds, this method gives a comprehensive evaluation, helping to reduce bias and ensure the model’s effectiveness on new and unseen data.

Table 2 compares the performance of one- and five-shot segmentation on the PASCAL-5i dataset. Our method achieves a significant 4.6% mIoU improvement over the baseline (SSP) in one-shot settings. The relatively moderate gain in one-shot, compared to five-shot, is likely due to the extreme scarcity of support information, which limits the multi-branch network’s ability to fully resolve background noise. However, in five-shot settings, our method achieves an overall mIoU of 73.8% (+4.5% over baseline). This superior performance demonstrates that with more comprehensive support data, our multi-branch architecture effectively leverages the complementarity between local textures and global semantic priors to achieve high-precision segmentation.

Table 3 presents the ablation study for the multi-branch configurations. While the dual-branch setup (ResNet and EfficientNet) shows a performance drop, this is likely due to feature redundancy, as both CNN-based backbones share similar inductive biases. However, the tri-branch network achieves a 70.0% mIoU, a 5.4% gain over the dual-branch version. This significant improvement demonstrates that the CLIP visual encoder provides essential global semantic priors that complement the local features of ResNet and EfficientNet. Consequently, the tri-branch architecture successfully achieves a more balanced and comprehensive feature representation.

In the one-shot setting, the model has access to only a single support example per class, providing minimal information for the multi-branch network to fully exploit feature complementarity. Consequently, while the tri-branch architecture still benefits from combining local and global features, the scarcity of support data limits its ability to resolve background noise and extract robust prototypes, resulting in a relatively moderate gain over the baseline. In contrast, the five-shot setting provides multiple support examples, offering more diverse local textures and global semantic priors. This setting allows the tri-branch network to more effectively leverage the complementary strengths of ResNet, EfficientNetV2 and the CLIP visual encoder. As reported in Tables 2 and 3, the five-shot configuration achieves a 4.5% mIoU improvement over the baseline, and our method outperforms the dual-branch setup by 5.4%, demonstrating that the additional CLIP branch significantly enhances feature representation when sufficient support information is available. This explanation clarifies that the limited improvement in one-shot is a natural consequence of the extreme scarcity of support data, rather than an inherent limitation of the tri-branch design. It highlights the network’s potential under more informative few-shot settings.

Table 4 lists the experimental results of the enhanced feature fusion and weighted mask method. In our study, features extracted by EfficientNetV2 and the CLIP visual encoder were concatenated and then subjected to a nonlinear transformation before being reduced to the same dimension as features of ResNet. Compared to the results without the enhanced feature fusion, this method yielded an additional 0.76% improvement. Feature fusion enables the capture of different levels and types of features, providing the model with more comprehensive data characteristics. In addition, leveraging the zero-shot generalization advantage of CLIP allows for more effective mitigation of noise impact. The experimental results of the weighted mask features indicate that incorporating these features enables the model to focus more on the foreground regions that need to be segmented. Compared to the results without the weighted mask features, incorporating them improves the model performance by 0.95%.

Figures 4 and 5 present the comparison between the proposed model and the baseline model on the one- and five-shot tasks, respectively. In addition, SCCAN (Xu et al., 2023a) is included for comparison. It can be observed that, compared to the baseline model, the segmentation area produced by the proposed model better highlights details. For instance, in the first image of the airplane, the baseline model fails to accurately segment the airplane wheels, whereas the proposed method can segment the wheels more accurately. Compared to the SCCAN model, the proposed model can more accurately match the contours of the ground truth, while SCCAN tends to segment background areas as foreground. Moreover, in the cases of birds, bottles and people, the proposed model not only better captures details but also provides more complete and ground truth-conforming segmentation areas.

In this study, we presented a novel few-shot semantic segmentation framework that leverages a multi-branch shared architecture to overcome the limitations of traditional feature extraction. By strategically combining the local feature encoding of ResNet and EfficientNet with the global semantic awareness of CLIP, the proposed model achieves a level of feature complementarity that single-backbone architectures lack. Our weighted mask and optimized fusion modules further refine this process, ensuring that the model remains sensitive to foreground targets while significantly reducing computational redundancy. The experimental success – characterized by a 4.6% and 4.5% improvement in one- and five-shot mIoU, respectively – validates the robustness of our approach. Beyond the numerical gains, this research provides a viable strategy for deploying deep learning models in “data-poor” specialized domains. Future work will focus on further optimizing the interaction between vision-language priors and structural features to enhance model generalizability across even more diverse image modalities.