ESAFormer: Multi-resolution Fusion Network for Pansharpening

With the development of satellite image sensors, the availability of remote-sensing images has increased in recent years. However, due to the technological limitations of existing sensors, images acquired by current remote sensing sensors have to make a trade-off between spectral and spatial resolution [20]. To meet the necessary signal-to-noise ratio (SNR), multispectral (MS) images with four or eight bands typically have a low spatial resolution, while PAN images tend to have a high spatial resolution but only consist of one band, as shown in Figure 1. However, a large number of high-resolution multispectral (HRMS) images are required for the interpretation of observed scenes. To create HRMS images, it is suggested to fuse low-resolution multispectral (LRMS) and PAN images using pansharpening [22].

Numerous pansharpening techniques have been proposed in the past few decades, with promising outcomes. Existing pansharpening can be divided into traditional methods and deep learning-based methods. The traditional methods include component substitution, multi-resolution, and model-based methods [20]. Component substitution and multi-resolution methods obtain fusion image from the perspective of detail injection [26]. The former method is to completely replace one component of the MS with a PAN image, such as the BDSD-PC [11]. The fusion results of the component substitution method have a good effect on the preservation of spatial details. However, due to the difficulty in separating the spectral and spatial information in the multi-spectral image, the spectral distortion of those results is often large. The latter method adopts the spatial details extracted from PAN images to integrate into MS images to obtain high-resolution multi-spectral images, such as the generalized Laplacian Pyramid (GLP) [1]. Multiresolution-based methods often achieve good results in spectral information retention. However, they model the spatial details of the source image and inject them into multispectral images, which tends to result in additional spatial artifacts in the fusion results.

Model-based methods focus on building feature representation models with appropriate regularizers to solve the fusion problem of multi-spectral and panchromatic images, then designing a module for the given model to reconstruct the high-resolution results [26]. The whole process can be regarded as the reconstruction of incomplete complementary observations of multichannel data from a mathematical point of view. In general, it mainly includes Bayesian, sparse reconstruction (SR), and model-based optimization (MBO) techniques. For example, the P+XS method obtains the spectral information of a fused image by assuming that the PAN image can be approximated as a linear combination of high-resolution multispectral bands [5]. However, model-based methods often require hyperparameter adjustment, and the calculation burden is heavy.

Since these traditional methods rely on manually created features [6][15], their capacity to fuse images to preserve features is severely constrained. Due to CNN-based methods having strong feature representation ability, many CNN-based pansharpening techniques are superior to traditional methods [7] [14]. In the early stage, a three-layer convolutional neural network (CNN) was designed to process the pansharpening task of multi-spectral images, and better fusion results were obtained compared with the traditional existing methods [19]. After that, a deep CNN structure was developed to fuse the PAN and LRMS images by high-pass filtering technology [30]. Deng et al. [7] designed Fusion-Net with the idea of component substitution and multiresolution. Recently, TF-Net [16] employed the deep convolution layer of residual connection to process the feature cascade, and performed image fusion in the feature domain. Zhang et al. designed TD-Net utilizing bi-directional information flow and multi-layer convolution processing to gradually generate fused images [32]. Nevertheless, CNN-based techniques still have certain drawbacks, such as the absence of long-range information modeling.

To address the lack of long-range dependency in CNN methods, Transformer-based pansharpening methods were developed and have achieved superior results in context presentation modeling [13, 26]. Dosovitskiy et al. firstly proposed a ViT-based approach for pansharpening with excellent results [9]. Subsequently, Meng et al. cut the source image into patches and used Transformer to learn the long-range dependency between these patches [21]. After that, Zhou et al. utilized reversible neural modules to represent and fuse features [35]. Zhang et al. [8] used the spatial and spectral attention mechanism to extract feature maps and used the graph attention mechanism to learn the similarity between feature maps. Although the global attention mechanism of Transformer allows the model to focus on feature dependencies over long distances, its ability is insufficient to represent local features that are very important in remote sensing image tasks.

In a word, the lack of synergy between long-range and short-range features at different scales in the above methods led to inadequate feature extraction and unsatisfactory pansharpening results. Meanwhile, the upsampled LRMS and PAN images are crudely layered in several deep-learning-based pansharpening approaches, while ignoring the resolution difference of the source images. As a result, the spatial features of the fused image are not significant enough.

To address the above issues, and inspired by recent work on image superresolution [10, 23], we propose a novel multi-resolution fusion network for pansharpening, named ESAFormer. The overall architecture of the network uses a bi-directional information flow of LRMS and PAN images, where the spatial and spectral features of MS and PAN images are integrated progressively. To preserve as much spatial detail as possible, we introduce the enhanced spatial attention mechanism that uses as few parameters as possible to achieve better spatial detail retention. In the MS branch, we merge CNN with Swin-Transformer’s attention mechanism to enable the model to combine their strengths in modeling both local and global information. The contribution of our work can be summarised as follows:

1) We propose a novel multi-resolution fusion network, which adopts a bi-directional structure to fully utilize the multi-resolution information of PAN image and spectral information of MS image.

2) We apply the enhanced spatial attention mechanism and Swin-Trans-former to train the model, which enables ESAFormer to obtain richer short and long-range features at different scales.

3) Experiments on multiple datasets show that ESAFormer outperforms SOTA methods in both visual and quantitative comparisons. Ablation studies further demonstrate the effectiveness of the proposed method.

In this section, we present the proposed method. Firstly, we give the problem formulation of the pansharpening task. Subsequently, the overall structure of the proposed approach is introduced. After that, we describe the implementation details of ESAFormer.

For convenience, the notation used throughout this paper is presented first. Let P ∈ ℝ^H×^W×¹ denotes a high-resolution PAN image with the spatial size of H × W and ms$∈ℝH4×W4×B$ denotes a low-resolution multi-spectral image with B spectral bands and spatial size $H4$⁠. To preserve the spectral information of MS images, the upsampled MS images M S ∈ ℝ^H×^W×^B are added to the final output. The whole process can be formulated as:

where Fused ∈ ℝ^H×W×B is the fused result, H_net denotes the whole network, and θ denotes the parameters.

The overall network architecture is a bi-directional information flow structure including MS and PAN branches as shown in Figure 2. The overall process is to extract the spatial information of PAN images and inject it into MS images progressively. The fused image is generated by integrating multi-spectral and spatial information from MS and PAN images gradually.

The PAN branch adopts two Conv Blocks consisting of four convolu-tional layers and a residual concatenation to generate different resolution spatial features. The MS branch consists of two steps of upsampling and ESAFormer, corresponding to different resolution features of PAN. In each step, the ESAFormer module fuses spectral and spatial features to generate the fusion results with the corresponding resolution. Finally, the HRMS image is obtained by adding the multi-resolution fusion result with the MS image after 4-fold upsampling.

This section introduces the workflow of ESAFormer, whose structure is shown in Figure 3. The ESAFormer module is the core module of the proposed network architecture. This module is adopted for integrating the spatial information of PAN and the spectral information of MS and generating the fused features.

Firstly, the upsampled MS image and the feature maps of PAN are directly stacked together to obtain the fused shallow features. Then, ESA is deployed to increase the receptive field of the model. After that, the intermediate features are fed into the Swin-Transformer Block to further integrate the spectral and spatial information of the whole fused features. Finally, the two-layer convolution generates the initial fusion results while avoiding excessive spatial artifacts. The process of the ESAFormer can be described as:

where H_STB(·) denotes a Swin-Transformer Block, Conv_group1(·) is the operation of the convolution layer group, and H_esa(·) denotes the ESA module. F_in denotes the shallow feature of the fused image, which could be formulated as :

where PAN_f denotes the feature maps of PAN image, m_up denote the upsam-pled MS image.

The enhanced spatial attention (ESA) module is shown in Figure 4. Given an input F_in, ESA firstly obtains features F_ı as follows:

where F₁ is to reduce embedding dimension, and Conv₁ is a 1 × 1 convolu-tion. Then ESA further calculates features F₂ as follows:

where Up(·) is the up-sampling function realized by bilinear interpolation, Pooling is a max-pooling operation, Conv_group2(·) is a group composed of two 3 × 3 convolution layers with Relu, and Conv₂ is a 3 × 3 convolution with stride of 2. Both the pooling layer and the stridden convolutional layer diminish the spatial dimensions, which are subsequently recovered by the upsampling layer. Finally, the output of ESA module can be computed as:

where Conv₃ is 1 × 1 convolutional layer used to recover the embedding dimension, Sigmoid is the sigmoid function, and symbol × denotes element-wise multiplication operation. The features are more narrowly concentrated on the areas of interest at the start of the ESAFormer thanks to the operation of the ESA mechanism. After this operation, we aggregate the key areas of the image and greatly retain and enhance the spatial details, which is conducive to the subsequent work of pansharpening.

The architecture of Swin-Transformer Block (STB) is shown in Figure 5. It consists of layer-normalization (LN), window-based multi-head self-attention (W-MSA), shifted W-MSA (SW-MSA), and position-wise multilayer perception (MLP). STB can be explained by the following formulas:

In above formulas, F₃, F₄, F₅, $Foutstb$ indicate the feature maps in different layers. It is worth noting that $Finstb$ equals to $Foutesa$⁠.

In the flow of Swin-Transformer [17], the input image is first passed through an embedding layer to convert the pixel values into an embedded representation that can be processed by the Transformer model. Second, the image is divided into a series of windows, and there is some overlap between these windows. The self-attention mechanism within each window is adopted to capture local features, while the cross-attention mechanism between the windows is employed to capture long-range context information. In cross-attention, the shifted-window strategy is adopted to further improve the information transmission. Layer Normalization and MLP components are applied to normalize and map features for better feature presentation. Finally, residual connections are utilized to retain and convey information.

At the end stage of ESAFormer, the spectral and spatial information in the fused feature is integrated through two-layer convolution processing.

In this section, we conducted extensive comparison and ablation experiments to validate the performance of the ESAFormer. Firstly, we introduce the two datasets employed in our experiments. Subsequently, implementation details and evaluation metrics are given separately. After that, we indicate and analyze the visual and quantitative comparison results with other SOTA methods. Finally, the results of the ablation study are presented.

WorldView-2 (WV2) and IKONOS (IK) datasets were adopted to verify the effectiveness of our method.

WorldView-2 (WV2):WorldView-2 dataset was captured from WorldView-2 satellite, which is the first commercial satellite to feature a Very-High-Resolution (VHR) sensor with more than four spectral bands. WV2 dataset has 500 pairs of images, each pair consisting of a PAN image with a resolution of 0.5 meters and a MS image with a resolution of 2 meters. The spatial sizes and radiometric resolutions of PAN/MS images are 1024 × 1024 / 256 × 256 and 11 bits respectively. The reduced-resolution (RR) data is obtained by degrading the MS and PAN images according to Wald’s methodology [29]. As there is no GT, the original MS images are then used as GT. In the reduced resolution test, we used the 8:2 distribution to set the training set and the test set. In the full-resolution test, we randomly selected 50 pairs of images to test.

IKONOS (IK):The IK consists of 200 pairs of images with 1 meter PAN images and 4 meters MS spatial resolutions. As same as WV2 dataset setting, we applied 8:2 distribution for training and testing. 40 pairs of images were selected for testing in our experiments.

Training Platform and Parameter Configuration: The proposed network was coded using Pytorch 1.7.0 and Python 3.8.0, and trained with an NVIDIA GPU GeForce RTX 3090. To minimize the loss function, we used the Adam optimizer with the betas of (0.9, 0.999) and weight decay of 0. We set the initial learning rate of 0.0004 and batch size of 4. To achieve better performance, every model underwent 1000 training epochs, with half-rate drop-in learning rate taking place every 200 epochs.

The similarity between the fused image and the ideal reference image (the original MS image) is measured by the reduced-resolution (RR) and full-resolution (FR) assessments. Multiple assessment metrics can be calculated to find the similarity. In the RR experiments, the Q2n (Q8 for 8-band datasets and Q4 for 4-band datasets) [4], the spectral angle mapper (SAM) [31], the dimensionless global error in synthesis (ERGAS) [29], and the spatial correlation coefficient (SCC) [34] are used. For SAM and ERGAS, the optimum values are 0, while for Q2n and SCC, they are 1. FR experiments are also required to validate the fusion performance. In contrast to the test scenarios with RR, there is no reference (GT) image. Therefore, three metrics without GT involved are applied, including the quality with no reference (QNR), the spectral distortion D_λ, and the spatial distortion D_s [25]. We use the residual standard error (RSE) as a measure of the difference in the spectra between the fused image and the GT.

To verify the effectiveness of the proposed method, extensive comparative experiments were conducted between ESAFormer and twelve state-of-the-art pansharpening methods over two datasets. To be specific, our method was compared with seven traditional methods, including EXP [2], BT-H [18], BDSD-PC [11], C-GSA [3], SR-D [24], MTF-GLP-HPM-R [27], and MTF-GLP-FS [28]. Five representative deep learning-based methods were selected, namely, PNN [19], DiCNN [12], Fusion-Net [7], TD-Net [32], and TF-Net [16].

WV2 RR:Figure 6 shows a scenario of urban buildings from WV2. For better visibility, we presented the RSE between GT and the fused images across the eight spectra as shown in Figure 7. In Figure 7, the lower RSE result proves the better performance. As can be seen from these two figures, traditional pansharpening methods have poor result, such as SR-D and C-GSA suffer from certain spectral distortion and spatial detail blurring. However, deep-learning-based methods retain the details well, such as Fusion-Net, TF-Net have less difference between GT and fused results than traditional methods. Compared to the above methods, the visualization results of our method have the best performance. Furthermore, we did quantitative comparison experiments to further illustrate the superiority of our method, as shown in Table 1 (Q8, SAM, ERGAS, and SCC). Our method achieved the best values in all 4 metrics in RR experiments. Especially in Q8, our method outperforms traditional methods over 5% and outperforms current deep learning methods such as TF-Net over 1%. The quantitative and visual results demonstrate the effectiveness of our method.

IK RR:Figure 8 is a scenario including different geographical instances such as land, terraces, and rivers. Similar to WV2 RR experiments, we presented the results of RSE for this scene, as shown in Figure 9. As can be seen in Figure 9, the RSE of traditional methods such as I-MTF-GLP-FS, I-MTF-GLP-HPM-R is higher, while the deep learning-based methods such as TD-Net, PNN is similarly higher. This means that their fusion is less effective. As shown in Table 2 (Q8, SAM, ERGAS, and SCC), our method achieved best values on all 4 metrics. Especially in ERGAS, our method significantly outperforms other methods. These quantitative and visual results prove the superiority of our method.

WV2 FR:Figure 10 shows the full resolution experiment performed on WV2 dataset. As shown in Figure 10, some methods have severe spectral distortion of the vegetation in BT-H, C-GSA, and PNN. Some spatial structure of fused images of BDSD-PC, MTF-GLP-HPM-R, and MTF-GLP-FS are lost. With full-resolution experiments at WV2, our method is the best in both spectral fidelity and spatial detail. As can be seen from Table 1 (D_λ, D_s, and QNR), our method achieved the best values for the QNR metric and ranks among the top of all methods for both the D_λ and D_s metrics. The above data fully demonstrate the superiority of our method in full-resolution experiments.

IK FR:Figure 11 presents the full-resolution experiments performed on the IK dataset. From the visualization results, we can see that the spatial details are slightly worse in the traditional methods such as BDSD-PC, SR-D, and MTF-GLP-FS. The spectral distortion is worse in PNN and Fusion-Net, but the overall results are still good in our method and TF-Net. As can be seen from Table 2 (D_λ, D_s, and QNR), our method achieved top-rank in all metrics, which demonstrates the excellent performance of our method in full-resolution experiments.

To verify the effectiveness of the components of our network, we did ablation study. Multiple experiments with different Network configurations were performed. Configurations include baseline [33], w/o ESA, ESA × 2, w/o branch, and ESAFormer.

baseline: We use a bi-directional flow network without ESAFormer module as our baseline. As shown in Table 3, our method is higher than the baseline in all the metrics. This greatly demonstrates that ESAFormer module has greatly improved our spectral preservation and spatial enhancement ability.

w/o ESA: In this experiment, we removed the ESA of the ESAFormer module and retain the original STB and convolutional layer structure. The experimental results reported that all the metrics of our method is higher than the model without ESA. Obviously utilizing the ESA to increase the receptive field is necessary for the pansharpening task. This further demonstrates that the proposed method is rational and effective.

ESA × 2: In this experiment, we explored whether multiple ESA components are necessary to achieve better results for the pansharpening task. We added an extra ESA after the STB in the ESAFormer module. The experimental results show that the model with additional ESA leads to performance decline instead. Therefore, although the strategy of using ESA to increase the receptive field is effective, too many ESAs stacked can lead to dilution of local features by overly dispersed contextual weights. Thus it is important to set an appropriate number of ESAs for the pansharpening task.

w/o branch: To verify the effectiveness of the bi-directional structure, we removed the PAN branch. We fed the stacked 4-fold upsampled MS and PAN images directly to the network instead of upsampling them step by step. From the results, we can see that the results of the single-branch structure on all the metrics are lower than our method. This suggests that the bi-directional structure plays an important role in the pansharpening task, in other words, the cascading injection of different resolution features is necessary.

In this paper, we propose a multi-resolution network that combines CNN and Transformer to perform a multi-scale fusion by a bi-directional structure. This structure makes full use of the spatial details of PAN images and gradually injects them into MS images. We innovatively proposed the ESAFormer module, which utilizes the enhanced spatial attention mechanism and the Swin-Transformer Block. In the proposed method, the spectral features can be better preserved and the spatial details can be enhanced. In comparison with other SOTA methods, the superiority of our method is demonstrated. Through ablation study, the effectiveness of the proposed components is verified.

This work was supported by the Fundamental Research Funds for the Central Universities (20199236855), the Aeronautical Science Foundation of China (D023030002), and the National Key Research and Development Program of China, project 3 (2022YFB4703703). We acknowledge the authors of WorldView-2 and IKONOS datasets used for this study.