Essence of image quality

Perceptual image quality has long been a central topic in computer vision and image processing. The field of image quality assessment (IQA) has made significant progress in measuring quality, both with reference (FR-IQA) and without reference (NR-IQA) (Mittal et al., 2012a; Sheikh and Bovik, 2006; Wang et al., 2004). Deep neural network-based IQA methods (Bosse et al., 2017; Kang et al., 2014; Ke et al., 2021; Zhang et al., 2018) have achieved high correlation with human subjective ratings, such as Mean Opinion Score (MOS). However, while IQA excels at quantifying overall image quality, it lacks mechanisms to decompose quality into contributing factors. This decomposition is inherently challenging because perceptual quality is subjective, context-dependent and entangled with semantic content – altering one aspect of an image (e.g. sharpness or noise) can unintentionally affect perceived content or style.

As illustrated in Figure 1(a), the relationship between individual image attributes (e.g. sharpness, noise, contrast) and perceptual quality is highly nonlinear, making the impact of each attribute on MOS inconclusive. Moreover, as shown in Figure 1(b), interactions among multiple attributes further complicate their joint influence on perceived quality. Together, these observations indicate that existing methods (Blau and Michaeli, 2018; Ke et al., 2021; Mittal et al., 2012a; Sheikh and Bovik, 2006; Wang et al., 2004; Zhang et al., 2018) – which typically measure only overall quality – cannot reveal which image attributes primarily drive perceptual judgments or how semantic content interacts with them. This underscores the difficulty of uncovering the essence of image quality.

Recently, text-to-image diffusion models (Ho et al., 2020; Rombach et al., 2022; Song et al., 2020) such as stable diffusion (SD) (Rombach et al., 2022) have demonstrated remarkable ability to generate high-fidelity, diverse and semantically coherent images. Beyond their generative capacity, these models support textual inversion (Gal et al., 2022; Ruiz et al., 2023), allowing optimization of a new token to represent a specific concept in embedding space. This capability opens the door to learning disentangled embeddings that capture perceptual quality independently of semantics. By leveraging the frozen backbone of SD and its text-conditioning mechanism, we can build an embedding space that explicitly separates semantic content from image quality.

In this work, we propose a novel framework for image quality decomposition, bridging IQA and generative modeling. Specifically, our approach first extracts semantic-preserving embeddings via textual inversion on SD, then learns a set of quality embeddings according to MOS differences. This enables not only controllable image synthesis but also a systematic study of the underlying essence of image quality, by disentangling perceptual factors from semantic content. To the best of our knowledge, this is the first work that systematically disentangles and controls image quality within powerful diffusion-based generative models.

Diffusion models have demonstrated impressive capabilities in image generation, yet controlling the generation process requires effective inversion methods. A naive approach to inversion is to add noise to an image and then denoise it through the network, but this often leads to significant changes in image content. To address this (Choi et al., 2021), conditions the denoising process on noised low-pass filter data from the target image, while Denoising Diffusion Implicit Mode (DDIM)-based methods (Dhariwal and Nichol, 2021) enable closed-form inversion to extract latent noise maps capable of reproducing a given real image. These techniques have been extended to enable cross-image interpolations and semantic editing, often using CLIP-based latent vectors for conditioning (Ramesh et al., 2021). However, such approaches are typically limited when generating novel conditions for a subject while maintaining fidelity.

In text-to-image diffusion models, inversion can be performed in two main latent spaces: the Image-based Noise Map (INM) space and the Textual Word Embedding (TWE) space (Dong et al., 2025; Gal et al., 2022; Kumari et al., 2023; Li et al., 2015; Qiu, 2016; Qiu et al., 2015, 2019b; Ruiz et al., 2023; Wang et al., 2018). INM-based methods, such as DDIM (Song et al., 2020) and Null Text (Mokady et al., 2023), recover the initial noise to faithfully reconstruct the image but generally suffer from limited editing capabilities. In contrast, TWE-based methods, including Textual Inversion (Gal et al., 2022) and DreamArtist (Dong et al., 2025), optimize the embedding of new pseudo-words to represent user-provided concepts, allowing for more flexible and intuitive textual editing. DreamBooth (Ruiz et al., 2023) further fine-tunes the entire model to learn high-fidelity concepts with unique identifiers, while Custom Diffusion (Kumari et al., 2023) improves efficiency by updating only key and value mappings in cross-attention layers. Despite these advances, learning a single new concept can still take several minutes, limiting scalability.

Textual inversion methods leverage the semantic richness of pre-trained text-to-image models to efficiently represent new concepts (Gal et al., 2022; Qiu et al., 2019a, 2020, 2021; Sun et al., 2021; Xu et al., 2022; Yang et al., 2020a, 2020b). By introducing new pseudo-words into the text embedding space, these methods enable personalized image generation through standard text prompts. Gradient-based optimization (Cui et al., 2025; Li et al., 2025; Ren et al., 2025a, 2025b; Ruder, 2016; Tang et al., 2026; Xu et al., 2024; Yu et al., 2024; Zhang et al., 2025a, 2025b) with calibrated learning rates ensures that the pseudo-word embeddings faithfully capture the target concept without catastrophic forgetting. This framework allows inversion techniques to not only reconstruct images but also provide a decomposition of visual concepts semantically. These capabilities of diffusion models provide a natural foundation for learning disentangled embeddings, making image quality decomposition feasible.

Extensive subjective experiments have been conducted to understand human perception of image quality, leading to the establishment of numerous image quality assessment (IQA) data sets. For instance, the LIVE database (Sheikh et al., 2006) collected subjective ratings for images degraded by JPEG, JPEG2000, white noise, Gaussian blur and simulated Rayleigh fading channel distortions under strictly controlled laboratory conditions. In addition to these controlled data sets, several large-scale IQA data sets have been created via crowdsourcing, such as TID2008 (Ponomarenko et al., 2009), KonIQ-10k (Hosu et al., 2020) and LIVE In the Wild (Ghadiyaram and Bovik, 2015). TID2008 contains 1,700 images covering a wide range of distortions, including noise, blur, compression artifacts, transmission errors and variations in brightness and contrast. KonIQ-10k was designed to capture real-world conditions, collecting 1.2 million subjective ratings from 1,459 participants across 10,073 images. The LIVE In the Wild data set comprises 1,162 images captured with diverse mobile devices, aiming to evaluate authentic distortions arising during capture, processing and storage. While databases such as TID2013 and KADID-10k focus on artificially introduced distortions, LIVE In the Wild emphasizes the evaluation of naturally occurring degradations.

Building upon these subjective studies, various computational IQA methods have been developed for quality prediction. Depending on the availability of reference images, these models are typically categorized as full-reference (FR), reduced-reference (RR) or no-reference (NR) IQA. Furthermore, IQA approaches can be divided into traditional methods relying on handcrafted features and deep learning-based methods that automatically learn relevant representations. Classical FR-IQA techniques include structural similarity index measure (SSIM) (Wang et al., 2004), MS-SSIM (Wang et al., 2003) and VIF (Sheikh and Bovik, 2006), whereas deep learning-based FR-IQA methods include DR-IQA (Ghildyal and Liu, 2022) and IQT (Yin et al., 2022). The proliferation of digital media has increased interest in NR-IQA, with traditional approaches often relying on natural scene statistics (Gao et al., 2013; Mittal et al., 2012a, 2012b; Moorthy and Bovik, 2010) or engineered features (Ma et al., 2017; Ye and Doermann, 2012; Ye et al., 2014, 2012).

Recently, deep learning has significantly advanced IQA performance. Convolutional neural network-based methods, such as WaDIQaM (Bosse et al., 2017) and DB-CNN (Zhang et al., 2020), have shown strong capability in extracting quality-relevant features. ResNet-based HyperIQA (Su et al., 2020) distinguishes between low- and high-level features and leverages high-level information to refine low-level representations. MetaIQA (Zhu et al., 2020) employs meta-learning to adapt networks to specific distortion types, enhancing prior knowledge utilization. Transformer-based approaches, including MANIQA (Yang et al., 2022) and DEIQT (Qin et al., 2023), exploit attention mechanisms for feature aggregation and efficient scoring, while StairIQA (Sun et al., 2023) integrates hierarchical features with iterative mixed-database training for improved generalization. More recently, AGAIQA (Wang et al., 2024) introduces adaptive graph attention and patch-wise hierarchical regression to further optimize feature exploitation and scoring precision.

However, despite these advances in perceptual quality assessment, no existing work has addressed the decomposition of image quality into factors and investigated whether semantics affect perceived quality.

We propose a framework that aim to investigate how different quality factors affect image perception by generating images with varying quality while preserving semantic content (Figure 2).

We build our approach upon latent diffusion models (LDMs) (Rombach et al., 2022), a recently proposed class of denoising diffusion probabilistic models (DDPMs) (Ho et al., 2020) that operate in the latent space of an autoencoder. LDMs comprise two main components. First, an autoencoder is pre-trained on a large image collection. An encoder E maps an image $x∈X$ to a latent code $z=E(x)$⁠, regularized via either a KL-divergence loss or vector quantization (Van Den Oord et al., 2017). The decoder D reconstructs images from latents such that $D(E(x))≈x$⁠.

The second component is a diffusion model trained to generate latents in the learned space. This model can be conditioned on class labels, segmentation masks, or outputs from a jointly trained text-embedding model. Let $cθ(y)$ denote a conditioning network that maps input y to a vector. The LDM loss is defined as:

where t is the diffusion timestep, $zt$ is the latent with noise added at step t, $ϵ$ is the sampled noise and $θ$ is the denoising network. Intuitively, the model learns to remove noise from latent representations. During training, both $cθ$ and $θ$ are optimized jointly to minimize equation (1). At inference, a random noise tensor is sampled and iteratively denoised to obtain $z0$⁠, which is then decoded into an image $x′=D(z0)$⁠. Besides, $cθ$ corresponds to a CLIP text encoder (Radford et al., 2021), with y as the text prompt.

Text encoder models, such as CLIP, tokenize an input string and map each token to a learned embedding vector. In our approach, we target this embedding space for concept inversion. Specifically, we introduce a placeholder string $S*$ representing a novel concept and replace its token embedding with a learnable vector $v*$⁠. This allows us to compose new prompts containing the concept as with standard words.

To learn $v*$⁠, we use a set of images depicting the target concept in varied poses and backgrounds. The concept embedding is optimized by minimizing the LDM loss over images from this set. Concept extraction is achieved using neutral context templates derived from CLIP ImageNet prompts (Radford et al., 2021), e.g. "A photo of$S*$". Specifically, the embedding inversion optimization is expressed as:

where the training procedure reuses the original LDM training scheme, keeping both $cθ$ and $θ$ fixed. As this is a reconstruction task, the learned embedding captures fine-grained details unique to the concept.

The influence of semantic content and low-level factors on image quality remains largely unknown. For instance, images with little or overly simple semantic content often receive lower subjective quality scores, while low-level factors such as blur or pixel noise also typically lead to lower quality ratings. However, the precise effects of these factors on image quality, as well as their potential interactions, are not yet fully understood. Therefore, in our methodology, we first fix the semantic content and then explore the impact of varying low-level factors on image quality. This approach allows us to better isolate and analyze the specific contributions of low-level factors when the semantic content remains constant.

Our goal is to disentangle semantic content from perceptual image quality, and to enable direct generation of quality levels while preserving semantics. We build upon a pre-trained SD model, which remains entirely frozen during training. The key idea is to introduce learnable tokens through textual inversion: one token represents the semantic content of an image, while another set of tokens captures different levels of image quality. We describe the process in three stages.

Given an image $Ii$ with a subjective quality score $mi$⁠, we first isolate its semantic content. To do so, we define a textual prompt of the form a photo of $S*$” where $S*$ is a learnable placeholder token. Using textual inversion, we optimize the embedding $vi*$ of $S*$ such that the frozen diffusion model can faithfully reconstruct $Ii$⁠. Concretely, we minimize the standard denoising loss:

where $z0,i$ is the VAE latent of $Ii$⁠, and $αt,σt$ denote the noise schedule. After convergence, $vi*$ provides a frozen semantic representation for image i, disentangled from its perceptual quality.

After obtaining $vi*$⁠, we further exploit the key property of classifier-free guidance, which is highly influenced by the unconditional prediction branch. Instead of modifying the model weights or conditional embeddings, we adopt null-text optimization (Mokady et al., 2023), which refines only the unconditional embedding $∅$⁠. Specifically, for each input image, $∅$ is initialized with the default null-text embedding and optimized while keeping the model parameters and conditional textual embedding fixed. This optimization yields a high-fidelity reconstruction of the input image while preserving editability through Prompt-to-Prompt (Hertz et al., 2022) by simply reusing the optimized unconditional embedding. Moreover, the optimized $∅$ can be reused for multiple editing operations after a single inversion, providing efficiency and consistency across edits. Since null-text optimization is inherently less expressive than model fine-tuning, it is performed under the pivotal inversion framework to ensure efficient and stable convergence.

Once the semantic token is fixed, we turn to modeling image quality. We discretize the MOS scores into K bins, each corresponding to a canonical quality level with center $μk$⁠. For each bin, we introduce a quality token $Xk$ with embedding $uk$⁠. To supervise these embeddings, we construct prompts of the form "a photo of$S*$with the image quality of$Xk$" as P, which jointly condition on semantics and quality information.

For a sample $Ii$ belonging to bin $ki$⁠, we optimize the diffusion loss:

This ensures that the model reconstructs $Ii$ when conditioned on the correct semantic embedding $vi*$ and its assigned quality embedding $uki$⁠.

Simply fitting each sample independently may not guarantee a coherent embedding space. We therefore impose a structure aligned with MOS scores. Specifically, we define a target similarity between quality embeddings and minimize as:

To further enforce relative ordering in the embedding space, we introduce a triplet margin loss based on MOS similarity. For each anchor embedding $uk$⁠, positive samples are drawn from MOS-similar bins and negative samples from MOS-dissimilar bins:

The triplet loss is then defined as:

In our implementation, MOS scores are discretized into $K=8$ equal-width bins, which is used to determine MOS-similar and MOS-dissimilar samples when constructing the positive and negative sets.

To avoid leakage of semantic information into quality tokens, we encourage orthogonality between the semantic embedding $vi*$ and all quality embeddings:

The full loss combines sample-wise reconstruction with global embedding structure:

where $λ⊥,λcos,λtri$ are balancing coefficients. Only the semantic tokens $vi*$ and the quality embeddings ${uk}$ are optimized; the diffusion model itself remains frozen.

At test time, given a target image $Itar$⁠, we first obtain its semantic embedding $vtar*$ using the procedure in equation (3). The user specifies a desired MOS value $mtar$⁠, which we map to a continuous quality embedding via soft interpolation:

The final conditioning is constructed by combining semantic and quality information. Concretely, we concatenate the semantic embedding $vtar*$ with the quality embedding $u˜(mtar)$ and feed them into the text-conditioning encoder together with the prompt "a photo of$vtar*$with the image quality of$u˜(mtar)$" to obtain the conditioning vector $ctar$⁠. We then use $ctar$ in the frozen diffusion model to generate $I^=D(z0)$⁠, which preserves the semantic content of $Itar$ while matching the desired perceptual quality. By varying the target MOS value $mtar$⁠, the model produces images with smoothly controllable quality levels.

To evaluate the effectiveness of our image quality decomposition framework, we conduct experiments on widely used image quality assessment benchmarks, including LIVE IQA (Sheikh et al., 2006) and TID2013 (Ponomarenko et al., 2015).

LIVE IQA data set (Sheikh et al., 2006) is a widely used benchmark for evaluating image quality assessment algorithms. It contains 29 reference images and 779 distorted images generated from these references using five common distortion types: JPEG compression, JPEG2000 compression, Gaussian blur, white noise and fast fading distortions. Each distorted image is associated with a MOS obtained from subjective experiments, providing a reliable measure of perceived image quality. Due to its diverse distortion types and well-calibrated subjective scores, LIVE IQA is extensively used for benchmarking image quality assessment methods.

TID2013 data set (Ponomarenko et al., 2015) is a large-scale IQA benchmark consisting of 25 reference images and 3,000 distorted images. The distortions cover 24 different types, including additive noise, blur, compression artifacts, color distortions and spatial transformations, each applied at five severity levels. Similar to LIVE, TID2013 provides MOS values obtained through extensive subjective evaluation. Its large variety of distortion types and levels makes it particularly suitable for evaluating the robustness of image quality generation and enhancement algorithms under diverse degradation conditions.

To quantitatively evaluate the performance of our image quality generation framework, we employ three widely used metrics: peak signal-to-noise ratio (PSNR), SSIM (Wang et al., 2004) and Learned perceptual image patch similarity (LPIPS) (Zhang et al., 2018). PSNR measures pixel-wise fidelity, SSIM evaluates perceptual similarity in terms of luminance, contrast, and structure, while LPIPS captures perceptual similarity using deep feature representations. Higher PSNR and SSIM values indicate better reconstruction quality, whereas lower LPIPS values indicate higher perceptual similarity.

We compare our method with three baseline approaches: Caption Prompt, Textual Inversion and DreamBooth.

Caption Prompt uses GPT-4 (Achiam et al., 2023) to generate textual descriptions that specify the target image quality. These prompts are fed into the pre-trained text-to-image model to decompose image quality.

Textual Inversion (Gal et al., 2022) learns new token embeddings for the target image quality using a small set of images. It can then generate new images reflecting the image quality with the learned embedding.

DreamBooth (Ruiz et al., 2023) fine-tunes the text-to-image model on a few images of the target image quality, enabling high-fidelity generation with strong identity preservation.

As shown in Table 1, our method achieves the best overall performance on both LIVE IQA and TID2013 across all three evaluation metrics. In particular, compared to DreamBooth, which already provides strong reconstruction quality, our approach yields further improvements in PSNR, SSIM, and LPIPS, indicating better fidelity and perceptual similarity. While Textual Inversion also enhances performance compared to Caption Prompt, its gains are limited due to the restricted expressive capacity of token-level embeddings. Caption Prompt, which relies solely on GPT-4 generated text descriptions, performs the worst since the quality guidance in natural language is often ambiguous and insufficient to capture fine-grained distortions. These results demonstrate that our method more effectively leverages textual conditioning for image quality generation, leading to consistent improvements under diverse degradation conditions.

As shown in Table 2, removing the quality embeddings leads to the most significant degradation, confirming that explicit quality tokens are essential for disentangling perceptual quality from semantic content. Without these embeddings, the model collapses to generic reconstructions with markedly lower PSNR and SSIM and higher LPIPS. Removing the cosine similarity loss also degrades performance, indicating that it plays a dominant role in structuring the quality embedding space to align with MOS-based perceptual distances and enabling smooth transitions across quality levels. In contrast, removing the triplet margin loss results in only a minor performance drop, as this component mainly refines class boundaries and enhances separability between distant quality levels rather than contributing substantially to reconstruction fidelity. Together, these results demonstrate that all three components are beneficial, with the cosine similarity loss and quality embeddings being the primary factors driving perceptual consistency and accuracy in our framework.

To further assess the effectiveness of our framework, we conducted a user study with 40 participants. Each participant was asked to complete two tasks. In the first task, they were presented with a reference image and four restored images produced by Caption Prompt, Textual Inversion, DreamBooth and our method. Participants were asked to select the restored image whose perceptual quality best matched the reference image’s MOS score. In the second task, participants compared image pairs generated by different methods and indicated which one appeared closer in perceptual quality to the reference.

As shown in Table 3, our method achieves the highest user preference in both tasks. Specifically, $38.2%$ of participants selected our results as most consistent with reference MOS scores in Task 1, and $34.2%$ preferred our method in direct pairwise comparisons. In contrast, DreamBooth ranked second but was consistently outperformed by our framework, while Caption Prompt and Textual Inversion lagged behind. These results corroborate the quantitative metrics and confirm that our approach provides perceptual advantages that are evident to human observers, consistent with prior findings on the importance of perceptual validation in image restoration (Sheikh et al., 2006; Wang et al., 2004; Zhang et al., 2018).

In this work, we introduced a novel framework for decomposing and analyzing perceptual image quality using text-to-image diffusion models. By leveraging textual inversion in SD, our method learns semantic-preserving embeddings alongside disentangled quality embeddings structured by perceptual metrics such as MOS. This enables systematic analysis of how perceptual quality is formed while maintaining semantic consistency, offering new insights into the essence of human visual perception. To our knowledge, this is the first approach to explicitly separate semantic and low-level perceptual factors using generative diffusion models. Future work will focus on improving cross-data set generalization, addressing differences in distortion types, severity distributions and subjective scoring across data sets. We also plan to extend the framework toward multidimensional perceptual embeddings that jointly capture distinct visual factors (e.g. sharpness, noise, contrast) for more robust and transferable modeling.

This research project was supported by International Communication Research Center of China Federation of Radio and Television Associations (2024C007).