Radiograph super-resolution with pixel-level masked D-MNet transformer

Medical imaging techniques, including radiograph, ultrasound, computed tomography (CT) (Hou et al., 2022) and magnetic resonance imaging (MRI), play a crucial role in disease screening, diagnosis (Lee et al., 2023), treatment planning and clinical evaluation (Suárez et al., 2023; Xie et al., 2023). Beyond the radiologist’s expertise, image quality is vital for precise diagnostic outcomes, particularly in X-ray imaging (Huang et al., 2022b). Low-resolution (LR) images often contain excessive noise, which can hinder the detection and analysis of critical structures. Consequently, radiologists may overlook or misdiagnose small tumors or lesions due to insufficient resolution. In contrast, high-resolution (HR) images offer enhanced clarity and detail, facilitating more accurate disease diagnosis and treatment planning for healthcare professionals. Single-image super-resolution reconstruction (SISR) would be desirable for solving this problem, by increasing the image resolution (Li et al., 2022; Wang et al., 2021b).

SISR (Chen et al., 2022; Huang et al., 2022a; Liu et al., 2022; Wang et al., 2020; Yu et al., 2023) aims to address the challenge of recovering HR images with enhanced detail from LR and HR image pairs. Recently, convolutional neural network (CNN)-based super-resolution (SR) models (Behjati et al., 2023, 2021; Dong et al., 2015, 2016; Lim et al., 2017; Sun et al., 2022; Zhang et al., 2018) have demonstrated remarkable performance in this domain. The advent of generative adversarial networks (GANs) has further advanced the field (Huang et al., 2021; Ledig et al., 2017). The integration of transformer architectures (Gao et al., 2022; Liang et al., 2021) into the SR discipline has garnered increasing interest from researchers due to their capacity to generate images with greater detail.

However, several challenges persist in the SR task for radiology images:

• First, X-ray images typically exhibit high contrast, where the edges and textural details of bones and other hard tissues are crucial for diagnosis. However, current SR baseline methods, primarily tailored for normal images, fall short in adequately addressing the intricate textural details when adapted to the X-ray images SR tasks.

• Furthermore, the small sample training data sets in real-world radiograph SR reconstruction tasks make neural network generalization capability limited.

In this study, we introduced a novel paradigm for SR tasks in radiometric images, called the depthwise (DW) convolution multiscale network Transformer: D-MNet Transformer (see Figure 1). Unlike previous methods, our proposed approach is dedicated to representing and extracting fine-grained local structures in X-ray images, such as skeletal features. Initially, local feature information is represented within the spatial features of each channel. Subsequently, pointwise convolution, using 1 × 1 convolution kernels, combines these features across channels (Chen et al., 2023), thus enhancing feature representation capabilities without losing local information. Traditional feature extraction modules based on conventional convolution often focus on blending local and global features in latent spaces, which negatively impacts the reconstruction performance. On the other hand, we integrate a UNet architecture within the feature extraction head to enhance the representation of shallow features. The UNet, with its multi-layered structure, captures image features at various scales and employs skip connections to directly map features from the encoder to the decoder, preserving the integrity of low-level features. This multiscale processing is particularly crucial for detail-rich X-ray images, where maintaining the fidelity of subtle features is essential for accurate analysis.

In addition, we explore the pixel-level masking strategy utilized in this pipeline. To the best of our knowledge, this is the first mask training method proposed for radiology images. This strategy strengthens model robustness for limited sample training data sets by randomly masking pixels in separate feature channels, without requiring additional convolution operations that would increase model consumption. Furthermore, our strategy follows the D-MNet principles reducing its dependency on neighboring pixels for local feature representation. This is achieved by employing random masking, which guides the model in generating a diverse range of latent feature representations.

The key contributions of this paper can be summarized as follows:

• We propose a shallow feature encoder, D-MNet, for radiological images. This module is specifically designed for the representation and extraction of fine local features in radiographic images. It adeptly uses spatial information representation within individual channels, coupled with pointwise convolution for seamless integration. In addition, the UNet framework further extracts features across multiple scales. Experimental results show that our encoder with separated channel information achieves better performance in the data distribution for a given task.

• We introduce a pixel-level masking strategy for medical radiology images to enhance generative capabilities with small sample training data sets. The proposed pixel-level masking strategy for the radiographic SR task allows introducing interference to enhance the robustness of the model with limited samples while preserving spatial information in key regions of the radiographic image. On the other hand, the implementation of a pixel-level masking strategy aids the model in reducing its dependency on neighboring pixels for local feature representation.

• We present a novel model for the radiographic SR task, the D-MNet Transformer, which incorporates the D-MNet based encoder and the transformer decoder. The utilization of a pixel-level masking strategy further improves the SR performance of the model. According to experimental results, our method generates radiometric images with more detailed information. It achieves state-of-the-art performance in PSNR metrics (upsampling factor: $×2$ and $×4$⁠), surpassing competing approaches.

SISR models:SISR is a classic vision task that aims to reconstruct an HR image from a given LR image. The introduction of deep learning-based approaches, which do not rely on carefully selected prior knowledge, has significantly advanced the field. Initially, shallow neural networks were introduced, surpassing the performance of traditional methods at the time (Dong et al., 2015, 2016). Subsequently, models with deeper structures were proposed to explore the potential of deeper networks in fitting data distributions. The incorporation of attention mechanisms further enhanced the models’ responsiveness to local and global features (Behjati et al., 2023; Lim et al., 2017; Zhang et al., 2018). Concurrently, the application of GANs to this task has garnered increased attention, with GANs (Ledig et al., 2017; Wang et al., 2021a, 2018) achieving remarkable results in recovering image details through an adversarial training strategy that deviates from traditional CNN-based models. Recently, the exceptional performance of transformer-based methods (Gao et al., 2022; Liang et al., 2021) in capturing global contextual information has also attracted growing interest in SR tasks. However, most of the popular methods focus on universal images. When these methods are applied to radiographic images, the model performance is limited because of the data domain shift. The principles of SR have also been successfully applied to various medical imaging modalities, such as enhancing CT image quality by jointly handling denoising and SR (Chi et al., 2024; Nagayama et al., 2025), or improving PET image analysis (Yu et al., 2025, 2024). Nonetheless, directly applying these models to radiographic images often yields suboptimal results due to unique domain characteristics. Therefore, it is necessary to propose models specifically designed for SR tasks in the given radiographic images.

Masking strategy: Drawing inspiration from masked language modeling in the field of natural language processing (Brown et al., 2020; Radford et al., 2018, 2019), the domain of computer vision has begun to explore masked image modeling (Bao et al., 2021; Dong et al., 2022; He et al., 2022), which is also regarded as a paradigm for self-supervised learning (Bachmann et al., 2022; Dong et al., 2022; Wang et al., 2023). The advantage of a masking strategy is its ability to enhance the robustness of CNN-based models, which proves beneficial for downstream tasks like object detection (He et al., 2017; Zhong et al., 2020). These strategies have also been demonstrated to be effective in low-level vision tasks, such as image restoration (Wang et al., 2023). In the radiographic SR task, the previous block masking strategy tends to lose too much spatial information because of the large masking area. Our pixel-level masking strategy enhances the model performance by considering the radiographic image channel information independently, and the pixel masking on the independent channels.

In this section, we will present the D-MNet used by this model for shallow local feature encoding and the pixel-level masking strategy applied to this specific task, respectively. Then, we will illustrate the transformer-based decoder for capturing global contextual information. Finally, we will also discuss the loss function used for model training.

For D-MNet, our objective is to enhance the capability of shallow feature extraction, specifically focusing on the representation and extraction of fine local structures within X-ray images, including skeletal formations. The D-MNet is constructed from a series of DW separable convolution blocks, a design choice motivated by both efficiency and the specific nature of radiographic images. Our core hypothesis is that the distinct anatomical components in radiographs – such as dense bone, soft tissue and air gaps – are best captured by specialized feature detectors.

Depthwise separable convolutions facilitate this by decoupling spatial filtering from cross-channel feature interaction. The DW convolution step allows each feature channel to learn an independent spatial filter, effectively becoming a specialized detector. Some channels may evolve to detect high-frequency bone boundaries and trabecular patterns, while others may specialize in modeling the low-frequency variations of soft tissue or identifying specific noise characteristics. The subsequent pointwise convolution then acts as a learned linear combination, intelligently fusing these disentangled feature maps to form a rich, multi-faceted representation. This separation prevents the network from learning “averaged” or overly generalized filters and instead encourages the creation of a diverse feature basis, which is crucial for high-fidelity reconstruction of complex medical imagery. This encoder-decoder structure, which first expands channel depth to a bottleneck and then symmetrically contracts it, forces the model to learn a powerful and compressed representation from this specialized feature set. As a fundamental component, the shallow feature encoder uses DW separable convolution (Chollet, 2017). The advantage of DW separable convolution is that it preserves more feature responses of the radiometric image on separate channels by concentrating on the convolution operation for each channel individually. According to the experimental results, respecting these separations more in the radiographic image SR task is beneficial. On the other hand, the DW separable convolution operation proves more beneficial for improving the reconstruction quality of SR images compared to the CNN-based local feature extraction module or the well-designed attention-based mechanism module (see Section 4.4). The proposed encoder incorporates a UNet structure, where multiscale feature extraction and interlayer skip connections further assist the model in concentrating on the diversity of scale in low-level features and maintaining the integrity of these foundational characteristics.

The detailed architecture of our D-MNet module is illustrated in Figure 2. The module is structured as an encoder-decoder network, composed of a sub-encoder (⁠$θD$⁠) and a sub-decoder (⁠$θE$⁠). The sub-encoder implements a multi-scale feature extraction strategy, processing the input through a series of depthwise separable convolution layers. As depicted, the feature channels progressively increase from 36- to a 1024-dimensional bottleneck and then symmetrically decrease, forcing the network to learn a compressed yet rich feature representation. The sub-decoder then utilizes a combination of depthwise separable convolutions and convolution transpose layers to reconstruct the feature representation back towards its original dimensionality. The entire D-MNet block is encompassed by a global skip connection, which adds the original input to the decoder’s output. This residual learning design, reminiscent of U-Net principles, is crucial for preserving high-frequency details and stabilizing the training process by allowing the module to focus on learning the modification rather than the entire transformation. By using feature channel scale magnification, the module is able to capture a more comprehensive range of pattern information throughout the spatial domain. The formalization for D-MNet can be described as follows: Let $C=c1,c2,…,cn$ be a set of channel numbers for nDW convolution operations. For the $ith$DW convolution operation, we obtain an intermediate feature map $fi$⁠:

where * denotes convolution, $Ik$ is the $kth$ channel of the input masked LR image $ILR′∼xlr′$⁠, $Ki,k$ is the $kth$ channel of the $ith$DW convolution kernel, and $(x,y)$ are the spatial coordinates in the feature map.

The D-MNet pseudo-code is also presented in Algorithm 1. Given a masked LR image $ILR ′∈RW′×H′×C′$⁠, where $RW′×H′×C′$ represents the number of channels and the spatial resolution, respectively. In our experiment, the value of C is 3. D-MNet initially extracts feature $Z0$ using a DW convolution operation with a filter size of $3×3$⁠. In the subencoder, we aim to optimize the parameter $θD$ by minimizing the risk. As illustrated in equation (2), $L$ denotes a loss function that depends on the parameter $θ$⁠. $IHR∼xhr$ refers to the HR image, and $E$ signifies the operation of taking the average for all $xlr ′$ in the mini-batch. Similarly, we seek to optimize the parameter $θE$ for the sub-decoder:

Algorithm 1: D-MNet pseudocode.

For the pixel-level masking strategy, our goal is to enhance the model’s performance on small sample training data sets by introducing perturbations to the channel feature maps. The pixel-level masking strategy also aids the model in reducing its reliance on neighboring pixels for local feature representation. Using random masks guides the model to generate diverse latent feature representations. Given that radiometric images exhibit dense local spatial domain responses (Huang et al., 2022b), we propose a masking strategy suitable for this specific SR task. Unlike previous block-based masking strategies (He et al., 2017; Zhong et al., 2020), our approach focuses on the pixel level. Based on experimental results, we found that block-based masking strategies can result in the loss of contextual information in the core region, making decoding more challenging. To address this issue, we perform masking operations on images at the pixel level. This approach offers the advantage of introducing perturbation to enhance the model’s robustness with limited training data, while minimizing the disruption to core spatial information. Moreover, this pixel-level masking strategy on feature channels also better preserves pattern information in radiometric images on separate channels, aligning with the design principles of D-MNet.

The pseudocode for the pixel-level masking strategy is shown in Algorithm 2. Given an LR image $ILR∈RW×H×C$⁠, where $W×H×C$ represents the number of channels and the spatial resolution, respectively. Using normal convolution, the feature map f with the number of channels $C′$ is obtained. Next, the random mask probability is determined by a given probability $pm∼N(0,1)$⁠. The mask operation is then carried out by traversing all the spatial pixels in f. Finally, the masked output $ILR′∈RW′×H′×C′$ is generated. This pixel-level masking is applied exclusively during the training phase. The rationale is to introduce a form of stochastic perturbation, compelling the model to learn more robust and generalizable feature representations from varied and partially obscured inputs, akin to a data augmentation strategy designed to improve performance on unseen data. During inference, the model receives the original, unmasked LR input to ensure deterministic and optimal reconstruction performance by utilizing all available information:

Algorithm 2: Pixel-level masking strategy.

In the D-MNet Transformer, the decoder’s goal is to decode latent spatial variables $Z0$ by capitalizing on the transformer’s ability to learn long-range dependent information. While CNN-based shallow image coding is proficient at capturing local spatial information, representing global long-distance information proves challenging due to the limited receptive field (Gao et al., 2022; Liang et al., 2021; Yoo et al., 2023). To address this issue, we turn to the transformer module from the field of natural language processing as a promising solution. Recent studies have demonstrated the effectiveness of the transformer, particularly when equipped with the distinctive self-attention mechanism and other enhancements, in downstream vision tasks. In this work, we opt for the recently proposed recursive transformer (Gao et al., 2022) as the decoder, given its balanced performance and computational efficiency.

Our framework can be approximated using neural networks by minimizing an $L1$ loss function:

The objective is to minimize the expected loss function $L1$ over the distribution of HR images $xhr$ and their corresponding SR images $xsr$⁠, represented by the joint probability distribution P. The loss function $L1$ quantifies the absolute difference between the $ISR$ and $IHR$ using the L1-norm (denoted by $∥ISR−IHR∥1$⁠).

Data sets. Our model is designed for radiographic SR tasks, specifically targeting small sample training data sets that closely resemble real-world scenarios. To achieve this, we utilized 500 samples from the publicly available MURA-SR data set (Huang et al., 2022b) to train our model and compare it with other baseline methods. This data set will also be available as a benchmark for small-sample radiographic SR tasks. For evaluation purposes, we adopted the test data sets from previous research on MURA-mini and MURA-plus (Huang et al., 2022b). These two test data sets share identical ground truth, but their LR images differ due to the application of varying degradation parameters.

Metrics. For comparison with other methods, PSNR and SSIM are chosen as our evaluation metrics. All evaluations will be completed in the Y channel of the YCbCr color space.

Experimental settings. The training details parameters and the hyperparameters of the model are shown in Table 1.

During the training phase, a cropping data augmentation technique is employed. Specifically, the network receives input images that have been cropped into $48×48$ patches. The learning rate is initialized at $2×10−4$⁠. Utilizing the PyTorch framework, the model is trained with the Adam optimizer on an NVIDIA RTX 3090 GPU. Utilizing the PyTorch framework, the model is trained with the Adam optimizer on an NVIDIA RTX 3090 GPU. Our training strategy involves two stages: we first pre-train the network using an L1 loss function to establish a stable baseline, and then fine-tune it using an MSE loss to further optimize for the PSNR metric.

For optimization, we adopt a two-stage training strategy designed to leverage the complementary strengths of different loss functions. In the first stage, we pre-train the network using the L1 loss. The L1 loss is known for its robustness to outliers and its ability to preserve sharp edges, making it ideal for the initial phase where the model learns the fundamental structural features of the radiographic images. By penalizing errors linearly, it prevents large initial prediction errors from destabilizing the training process and guides the model to a structurally sound solution.

In the second stage, we fine-tune the model using the MSE loss. After the model has learned the coarse features, the MSE loss is employed to refine the results for high-fidelity reconstruction. Because MSE Loss penalizes larger errors quadratically, it strongly encourages the minimization of all pixelwise discrepancies, which directly optimizes for the PSNR metric. This two-stage approach first establishes a robust baseline and then polishes the details, providing a clear and effective path to achieving state-of-the-art performance. The Adam optimizer is used throughout both stages with a learning rate of 2 × 10⁻⁴.

To evaluate the performance of the D-MNet transformer, we compared the proposed method with several benchmark methods, which also include the state-of-the-art models: SRCNN (Dong et al., 2015), FSRCNN (Dong et al., 2016), SRGAN (Ledig et al., 2017), EDSR (Lim et al., 2017), RCAN (Zhang et al., 2018), SwinIR (Liang et al., 2021), OverNet (Behjati et al., 2021), ShuffleMixer(base and tiny) (Sun et al., 2022), LBNet (Gao et al., 2022) and DiVANet (Behjati et al., 2023).

Table 2 presents a comparison of the D-MNet Transformer with 10 state-of-the-art learning-based methods. Notably, ShuffleMixer, LBNet, and DiVANet represent the most recent advancements in the field. The comparison includes not only conventional benchmarks but also an evaluation of lightweight methods. The experimental results indicate that our method achieves the best PSNR metric (30.43 and 30.82 dB) across all tested scenarios. While our SSIM scores are highly competitive (0.9509 and 0.9439 at $×4$⁠), we note that in some specific cases, other methods like SwinIR achieve marginally higher SSIM values (e.g. 0.9514 vs 0.9509 on ×4 MURA-mini). This suggests a potential trade-off between optimizing for different image quality metrics. Our model, particularly with its L1/MSE loss optimization strategy and focus on fine-grained feature separation, appears to prioritize pixelwise accuracy (measured by PSNR) over perceptual structural similarity (measured by SSIM). This trade-off is an interesting characteristic of our proposed architecture and highlights the ongoing challenge in developing a single model that universally excels across all evaluation metrics. Furthermore, competitive results are observed for the SSIM (0.9422 and 0.9333). This evidence suggests that our model is well-suited for data distribution in medical radiology image SR tasks. The effectiveness of our model is attributed to two primary factors: the D-MNet’s shallow feature representation, which respects the independence of feature channel information, and the pixel-level masking strategy that facilitates network training in the face of small sample challenges. More details will be discussed in the ablation experiments section. Next, we will show more details about the images in the qualitative result section.

In this section, the SR images at $×2$ and $×4$ upsampling factors are presented (see Figures 3 and 4). Experimental results indicate that our method excels in reconstructing edge texture information within the images, enabling the recovery of critical details, such as sharp edges, which are essential for diagnostic decisions. To facilitate a clearer understanding of edge detail recovery, Figures 5 and 6 present heat maps of the histogram of oriented gradients (HOG), visualizing the magnitude of edge gradients in the SR images. The accompanying color bar in these figures indicates the gradient strength, where colors transitioning from blue (low magnitude, ≈0) to yellow and red (high magnitude, ≈300) represent increasing edge intensity. Consequently, regions rendered in warmer colors (yellows and reds) signify stronger and more pronounced edge details. These visualizations suggest that our proposed method more effectively reconstructs fine textures and sharp edges, as evidenced by the gradient patterns and intensities in our results more closely resembling those of the HR reference compared to other methods. The utilization of HOG features for this analysis is not only effective for visualizing such edge information but also highlights their broader utility in computer vision tasks like object detection, where robust edge representation is critical.

The D-MNet Transformer incorporates two key concepts: first, it respects the independence between feature channel information when encoding shallow local features, which is accomplished through the use of DW convolution in the D-MNet architecture; second, it employs a pixel-level masking strategy between feature channels to enhance the model’s performance when dealing with small sample training data. In this section, we will present the corresponding ablation experiments.

As shown in Table 3, we evaluated various shallow local feature extraction blocks. To establish a clear comparative context, we included several modules with relatively fundamental or representative architectures, such as the Attention Head and the standard CNN-base, which serve as basic benchmark models to help quantify the gains from more sophisticated designs. Our Shallow D-MNet (30.00 dB), which consists of three DW convolution blocks, is specifically designated as the baseline for this set of experiments. In contrast, our proposed Deep D-MNet, which demonstrates superior model performance (30.43 dB), utilizes 9 DW convolution blocks. From the results, we can observe that modules primarily based on conventional CNN structures, for instance, the Attention Head (29.91 dB, whose design might draw upon principles from approaches like (Zhang et al., 2022)) and the CNN-base module (30.06 dB), are less effective for achieving optimal results on the data distribution of medical radiological images. In contrast, our proposed Deep D-MNet, which acts as a DW convolutional head in this context, demonstrates superior model performance (30.43 dB) compared to the module constructed with standard CNNs (30.06 dB). This advantage can be attributed to the operational nature of DW convolution, where most computations are performed on independent feature channels, thereby better preserving the distinct information within each channel. Furthermore, the experiments (comparing Deep D-MNet at 30.43 dB with the baseline Shallow D-MNet at 30.00 dB) also indicate that increasing the number of DW convolution layers is beneficial for the model’s performance.

For different shallow feature extraction modules, the PSNR line plots are shown in Figure 7(a). Our method can achieve the best performance compared to other baseline methods (Transformer and CNN or Only Transformer). On the other hand, the t-SNE (Van der Maaten and Hinton, 2008) method for dimensionality reduction visualizes the relationship between the SR samples reconstructed by our method and the HR images (see Figure 7(b) and (c). Our method demonstrates a better fit to the data distribution at scaling factors: $×2$ and $×4$⁠.

To understand the rationale behind our masking strategy, we first examine the characteristics of feature activations in radiographic images. As illustrated by the feature activation map in Figure 8(a), generated using CAM (Selvaraju et al., 2017), critical diagnostic information in radiographic images often concentrates in highly localized regions (e.g. the area highlighted by the red bounding box). Traditional block masking strategies, visualized in Figure 8(c), operate by obscuring contiguous patches of the input. If such a block were to cover a crucial localized activation area [as exemplified by comparing the potential impact of the masked region in Figure 8(c) with the active region in Figure 8(a)], important spatial information within that entire critical area could be completely lost. This coarse-grained masking can be detrimental when precise local details are paramount.

In contrast, our proposed pixel-level masking strategy, visualized in Figure 8(b), offers a more nuanced approach. While Figure 8(b) depicts the visual outcome of such a mask, the core advantage lies in its application: we apply these pixel masks independently across different feature channels. This means that even if a specific pixel is masked in one feature channel, its information might still be preserved and processed in other unmasked channels. This fine-grained, channelwise independent masking encourages the model to learn more robust and distributed representations, reducing the risk of catastrophic information loss in critical local areas compared to block masking.

Furthermore, we investigated the influence of the mask rate “pm” on the efficacy of the pixel-level masking strategy, with the results presented in Table 4. The study reveals a clear optimal value: a mask rate of $pm=0.5$ achieves the best performance (30.43 dB), significantly improving upon the baseline. This suggests that a 50% masking rate strikes an effective balance, providing a sufficient challenge for the model to learn robust features through reconstruction while retaining enough unmasked information for effective learning. When the rate is too high (e.g. 0.75), performance degrades, likely due to excessive information loss.

Based on this analysis, we concur that the optimal masking rate is likely a task-dependent hyperparameter. The ideal ratio may vary depending on factors such as the data set’s complexity, the degradation level (i.e. scaling factor), and the model’s capacity. Therefore, while $pm=0.5$ serves as a strong baseline for radiographic SR, we recommend that it be treated as a tunable hyperparameter when adapting our method to new applications or data sets to achieve the best possible performance.

In this work, we explore a new paradigm for the radiographic image SR task: D-MNet Transformer. First, the D-MNet feature extraction head enhances the representation of edges and textural details in X-ray images, being particularly sensitive to local domains and incorporating multiscale feature extraction. Furthermore, global contextual information is captured in the decoder using a transformer. We also propose a pixel-level masking strategy, which significantly enhances the performance of the D-MNet Transformer when trained with small samples, compared to the classical block masking strategy. We evaluate the model’s performance on benchmark data sets and achieve state-of-the-art results in the quantitative metric PSNR comparison.

This work was supported by JSPS KAKENHI Grant Number JP25K03130.