# boosting_latent_diffusion_with_perceptual_objectives__8af3b256.pdf

Published as a conference paper at ICLR 2025

BOOSTING LATENT DIFFUSION WITH PERCEPTUAL OBJECTIVES

Tariq Berrada Ifriqi1,2, Pietro Astolfi1, Melissa Hall1, Marton Havasi1, Yohann Benchetrit1, Adriana Romero-Soriano1,3,4,5, Karteek Alahari2, Michal Drozdzal1, Jakob Verbeek1

1 FAIR at Meta, 2 Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK, France 3 Mc Gill University, 4 Mila, Quebec AI institute, 5 Canada CIFAR AI chair tariqberrada@meta.com

A kitten with its paws up. A stone wall leads up to a solitary tree at sunset. Statue of a beautiful maiden hidden among the pretty pink flowers. Peach and cream brides bouquet for a rustic vintage wedding.

Figure 1: Samples from models trained with and without our latent perceptual loss on CC12M. Samples from our model with latent perceptual loss (bottom) have more detail and realistic textures.

Latent diffusion models (LDMs) power state-of-the-art high-resolution generative image models. LDMs learn the data distribution in the latent space of an autoencoder (AE) and produce images by mapping the generated latents into RGB image space using the AE decoder. While this approach allows for efficient model training and sampling, it induces a disconnect between the training of the diffusion model and the decoder, resulting in a loss of detail in the generated images. To remediate this disconnect, we propose to leverage the internal features of the decoder to define a latent perceptual loss (LPL). This loss encourages the models to create sharper and more realistic images. Our loss can be seamlessly integrated with common autoencoders used in latent diffusion models, and can be applied to different generative modeling paradigms such as DDPM with epsilon and velocity prediction, as well as flow matching. Extensive experiments with models trained on three datasets at 256 and 512 resolution show improved quantitative with boosts between 6% and 20% in FID and qualitative results when using our perceptual loss.

1 INTRODUCTION

Latent diffusion models (LDMs) (Rombach et al., 2022) have enabled considerable advances in image generation, and elevated the problem of generative image modeling to a level where it has

Published as a conference paper at ICLR 2025

become available as a technology to the public. A critical part to this success is to define the generative model in the latent space of an autoencoder (AE), which reduces the resolution of the representation over which the model is defined, thereby making it possible to scale diffusion methods to larger datasets, resolutions, and architectures than original pixel-based diffusion models (Dhariwal & Nichol, 2021; Sohl-Dickstein et al., 2015).

To train an LDM, all images are first projected into a latent space with the encoder of a pre-trained autoencoder, and then, the diffusion model is optimized directly in the latent space. Note that when learning the diffusion model the AE decoder is not used the diffusion model does not receive any training feedback that would ensure that all latent values reachable by the diffusion process decode to a high quality image. This training procedure leads to a disconnect between the diffusion model and the AE decoder, prompting the LDM to produce low quality images that oftentimes lack high frequency image components. Moreover, we note that the latent spaces of pre-trained LDM s autoencoders tend to be highly irregular, in the sense that small changes in the latent space can lead to large changes in the generated images, further exacerbating the autoencoder-diffusion disconnect problem.

In this work, we propose to alleviate this autoencoder-diffusion disconnect and propose to include the AE decoder in the training objective of LDM. In particular, we introduce latent perceptual loss (LPL) that acts on the decoder s intermediate features to enrich the training signal of LDM. This is similar to the use of perceptual losses for image-to-image translation tasks (Johnson et al., 2016; Zhang et al., 2018), but we apply this idea in the context of generative modeling and use the feature space of the pre-trained AE decoder rather than that of an external pre-trained discriminative network. Our latent perceptual loss results in sharper and more realistic images, and leads to better structural consistency than the baseline see Figure 1. We validate LPL on three datasets of different sizes the commonly used datasets Image Net-1k (1M data points) and CC12M (12M data points), and additionally a private dataset S320M (320M data points) as well as three generative models formulation DDPM (Ho et al., 2020) with velocity and epsilon prediction, and conditional flow matching model (Lipman et al., 2023). In our experiments, we report standard image generative model metrics such as FID (Heusel et al., 2017), CLIPScore (Hessel et al., 2021), as well as Precision and Recall (Sajjadi et al., 2018; Kynkäänniemi et al., 2019). Our experiments show that the use of LPL leads to consistent performance boosts between 6% and 20% in terms of FID. Our qualitative analysis further highlights the benefits of LPL, showing images that are sharp and contain high-frequency image details.

In summary, our contributions are: We identify a slight disconnect between latent and pixel-space diffusion which can lead to suboptimal results when training latent diffusion and flow models. We propose the latent perceptual loss (LPL), a perceptual loss variant leveraging the intermediate feature representation of the autoencoder s decoder. We present extensive experimental results on the Image Net-1k, CC12M, and S320M datasets, demonstrating the benefits of LPL in boosting the model s quality by 6% to 20% in terms of FID. We show that LPL is effective for a variety of generative model formulations including DDPM and conditional flow matching approaches.

2 RELATED WORK

Diffusion models. The generative modeling landscape has been significantly impacted by diffusion models, surpassing previous state-of-the-art GAN-based methods (Brock et al., 2019; Karras et al., 2019; 2020; 2021). Diffusion models offer advantages such as more stable training and better scalability, and were successfully applied to a wide range of applications, including image generation (Chen et al., 2024; Ho et al., 2020), video generation (Ho et al., 2022b; Singer et al., 2023), music generation (Levy et al., 2023; San Roman et al., 2023), and text generation (Wu et al., 2023). Various improvements of the framework have been proposed, including different schedulers (Lin et al., 2024; Hang & Gu, 2024), loss weights (Choi et al., 2022; Hang et al., 2023b), and more recently generalizations of the framework with flow matching (Lipman et al., 2023). In our work we evaluate the use of our latent perceptual loss in three different training paradigms: DDPM under noise and velocity prediction, as well as flow-based training with the optimal transport path.

Published as a conference paper at ICLR 2025

Latent diffusion. Due to the iterative nature of the reverse diffusion process, training and sampling diffusion models is computationally demanding, in particular at high resolution. Different approaches have been explored to generate high-resolution content. For example, Ho et al. (2022a) used a cascaded approach to progressively add high-resolution details, by conditioning on previously generated lower resolution images. A more widely adopted approach is to define the generative model in the latent space induced by a pretrained autoencoder (Rombach et al., 2022), as previously explored for discrete autoregressive generative models (Esser et al., 2021). Different architectures have been explored to implement diffusion models in the latent space, including convolutional UNetbased architectures (Rombach et al., 2022; Podell et al., 2024), and more recently transformer-based ones (Peebles & Xie, 2023; Chen et al., 2024; Gao et al., 2023; Esser et al., 2024) which show better scaling performance. Working in a lower-resolution latent space accelerates training and inference, but training models using a loss defined in the latent space also deprives them from matching highfrequency details from the training data distribution. Earlier approaches to address this problem include the use of a refiner model (Podell et al., 2024), which consists of a second diffusion model trained on high-resolution high-quality data that is used to noise and denoise the initial latents, similar to how SDEdit works for image editing (Meng et al., 2022). Our latent perceptual loss addresses this issue in an orthogonal manner by introducing a loss defined across different layers of the AE decoder in the latter stages of the training process. Our approach avoids the necessity of training on specialized curated data (Dai et al., 2023), and does not increase the computational cost of inference.

Perceptual losses. The use of internal features of a fixed, pre-trained deep neural network to compare images or image distributions has become common practice as they have been found to correlate to some extent with human judgement of similarity (Johnson et al., 2016; Zhang et al., 2018). An example of this is the widely used Fréchet Inception Distance (FID) to assess generative image models (Heusel et al., 2017). Such perceptual distances have also been found to be effective as a loss to train networks for image-to-image tasks and boost image quality as compared to using simple ℓ1 or ℓ2 reconstruction losses. They have been used to train autoencoders (Esser et al., 2021), models for semantic image synthesis (Isola et al., 2017; Berrada et al., 2024) and super-resolution (Suvorov et al., 2022; Jo et al., 2020), and to assess the sample diversity of generative image models (Schönfeld et al., 2021; Astolfi et al., 2024). In addition, recent works propose variants that do not require pretrained image backbones (Amir & Weiss, 2021; Czolbe et al., 2020; Veeramacheneni et al., 2023). An et al. (2024); Song et al. (2023) employed LPIPS as metric function in pixel space to train cascaded diffusion models and consistency models, respectively. More closely related to our work, Kang et al. (2024) used a perceptual loss (E-Latent LPIPS) defined in latent space to distill LDMs to conditional GANs, but used a separate image classification network trained over latents rather than the autoencoder s decoder to obtain the features for this loss. Lin & Yang (2024) added a perceptual loss (Self-Perceptual) to train LDMs that is defined over the features of the denoiser network in the case of UNet architectures. However, they found this loss to be detrimental when using classifierfree guidance for inference. In summary, compared to prior work on perceptual losses, our work is different in that (i) LPL is defined over the features of the decoder that maps from latent space to RGB pixel space, rather than using a network that takes RGB images as input and (ii) our LPL can be regardlessly applied to train both latent diffusion and flow models and has no architecture constraint in the denoiser model.

3 USING THE LATENT DECODER TO DEFINE A PERCEPTUAL LOSS

In this section, we analyze the impact of the decoder-diffusion disconnect on the LDM training, and then, we follow with the definition of our latent perceptual loss.

3.1 LATENT DIFFUSION AND THE ℓ2 OBJECTIVE

We use Fβ to refer to an autoencoder that consists of two modules. The encoder, F e β, maps RGB images x0 RH W 3 to a latent representation z0 RH/d W/d C, where d is the spatial downscaling factor, and C the channel dimension of the autoencoder. The decoder, F d β , maps from the latent space to the RGB image space. In LDM, the diffusion model, DΘ, with parameters Θ is defined over the latent representation of the autoencoder. We follow a typical setting, see e.g. (Peebles & Xie, 2023; Chen et al., 2024; Rombach et al., 2022), where we use a fixed pre-trained autoencoder with a downsampling factor of d = 8 and a channel capacity of C = 4.

Published as a conference paper at ICLR 2025

Training Objective. The diffusion formulation results in an objective function that is a lower-bound on the log-likelihood of the data. In the DDPM paradigm (Ho et al., 2020), the variational lower bound can be expressed as a sum of denoising score matching losses (Vincent, 2011), and the objective function can be written as L = P

t Lt, where Lt = DKL [q (xt 1|xt, x0) ||pθ (xt 1|xt)] =

Ex0,ϵ,t h βt (1 βt)(1 αt) ϵ ϵθ(xt, t) 2i . Ho et al. (2020) observed that disregarding the time step specific weighting resulted in improved sample quality, and introduced a simplified noise reconstruction objective, known as epsilon prediction, where the objective is the average of the MSE loss between the predicted noise and the noise vector added to the image, Lsimple = P

t Ex0,ϵ,t ϵ ϵθ(xt, t) 2 . The underlying idea is that the better the noise estimation, the better the final sample quality. An equivalent way to interpret this objective is through reparameterization of the target to the original latents, Lsimple = P

t λt Ex0,ϵ,t x0 ˆx0(xt, t; θ) 2 , where ˆx0(xt, t; θ) = (xt σtˆϵt)/αt and λt = 1/σ2 t .

We note that the presence of ℓ2 in the LDM objective has two important implications. First, the ℓ2 norm treats all pixels in the latents as equally important and disregards the downstream structure induced by the decoder whose objective is to reconstruct the image from its latents. This is problematic because the autoencoder s latent space has a non-uniform structure and is not equally influenced by the different pixels in the latent code. Thus, optimizing the ℓ2 distance in the diffusion model latent space could be different from optimizing the perceptual distance between images. Second, while an ℓ2 objective is theoretically justified in the original DDPM formulation, generative models trained with an ℓ2 reconstruction objective have been observed to produce blurry images, as is the case e.g. for VAE models (Kingma & Welling, 2014). Such an effect can lead to exposure bias in diffusion (i.e. denoiser input mismatch between training and sampling), which has been studied through the lens of the sampling algorithm but not the training objective Ning et al. (2024). The problem of blurry images due to ℓ2 reconstruction losses has been addressed through the use of perceptual losses such as LPIPS (Zhang et al., 2018), which provide a significant boost to the image quality in settings such as autoencoding (Esser et al., 2021), super-resolution Ledig et al. (2017) and image-to-image generative models (Isola et al., 2017; Park et al., 2019). To address these two implications of the latent ℓ2 optimization, we design a new loss, namely Latent Percetual Loss (LPL), that directly tackles the mismatch between the structures of the latent and the pixel space by matching the target and the predicted latents when partially decoded with the AE decoder. Compared to existing perceptual losses, e.g., LPIPS, that usually rely on features from pre-trained classifier networks, the use of AE decoder features is not standard. However, for LDMs training, the AE decoder provides a perceptually meaningful signal as the intermediate features of the decoder, being closer to pixel space, have a structure more similar to the pixel space. Moreover, these features have higher resolution than the diffusion latent features, which helps in modeling high-level details that reduce blurriness of the predicted image.

3.2 LATENT PERCEPTUAL LOSS

We propose a loss function that operates on the features at different depths in the autoencoder s decoder, F d β . Let zt = αtz0 + σtϵt be a noisy sample in the diffusion model latent space at time t, and ˆz0 = (zt σt D(zt, t; Θ))/αt the corresponding estimated noise-free latent at time t = 0. To enhance the accuracy of the reconstruction process in image space, we propose augmenting the diffusion objective with a penalty term that encourages the predicted forward process distribution to match the reconstructions ˆz0 at a lower level of compression. This is achieved by introducing an image-level projection penalty on the forward process, which can be expressed as:

Lpen t 1 = Eq [DKL (q(xt 1|xt, x0) pΘ(ˆxt 1|ˆxt))] . (1)

In Appendix A.1, we show that such a penalty, under certain conditions, can be approximated by computing a pairwise distance between the outputs of the projection decoder, which maps latents into samples in image-space. Guided by this result, our loss term should resemble a reconstruction term between the original image and the predicted/denoised image obtained from the decoder of the autoencoder. However, this does not address the problem of blurriness exposed in the previous section, which is why we opt for a loss structure that is similar to LPIPS, where the loss term is defined over the intermediate feature spaces of the decoder.

Published as a conference paper at ICLR 2025

ℒ! = 𝑧" 𝑧 " #

ℒ# = 𝜙#. / 𝑤$

ℒ# = 𝜙#. / 𝑤$

Figure 2: Overview of our approach. (a) Latent diffusion models compare clean latents and the predicted latents. (b) Our LPL acts in the features of the autoencoder s decoder effectively aligning the diffusion process with the decoder. F e β, F d β : autoencoder encoder and decoder, DΘ: denoiser network, CN: cross normalization layer, OD: outlier detection.

Framework αt σt ˆx0

DDPM-ϵt αt 1 αt xt σt D(xt, t; Θ) /αt DDPM-vθ αt 1 αt αtxt σt D(xt, t; Θ)

Flow-OT 1 t t xt σt D(xt, t; Θ)

Table 1: Summary of the formula for the estimate of the clean image corresponding to the different formulations. Using the following parameterization, t, xt = αtx0 + σtϵt.

We compute two sets of hierarchies of L decoder features, {ϕl}L l=1, and, {ˆϕl}L l=1, by decoding both the original, z0, and estimated latents, ˆz0 (where for brevity we drop the dependence of ˆz0 on t): ( ϕ1, ...,ϕL = F d,l β (z0)

l [[1,L]], ˆϕ1, ..., ˆϕL = F d,l β (ˆz0)

l [[1,L]]. (2)

Using these intermediate features, we can define our training objective. Our LPL, LLPL, is a weighted sum of the quadratic distances between the feature representations at the different decoding scales, obtained after normalization:

LLPL = Et T ,ϵ N(0,I),x0 DX

ρl,c(ˆϕl,c) ϕ l,c ˆϕ l,c 2 2

where ϕ l is the standardized version of ϕl across the channel dimension, ρl,c(ˆϕl,c) is a binary map masking the detected outliers in the feature map ˆϕl,c, ωl is a depth-specific weighting and Cl the channel dimensionality of the feature tensor. Note that we better explain these terms later in this section. Moreover, to reduce both LPL computational complexity and memory overhead, we only apply our loss for high signal-to-noise ratios (SNR). In particular, we impose a hard threshold τσ and only apply the loss if the SNR is higher than it δσt τσ(σt). The LPL loss is applied in conjunction with the standard diffusion loss, resulting in the training objective Ltot = LDiff + w LPL LLPL.

Depth-specific weighting. Empirically, we find the loss amplitude at different decoder layers to differ significantly it grows with a factor of two when considering layers with a factor two increase resolution. To balance the contributions from different decoder layers, we therefore weight them by the inverse of the upscaling factor w.r.t. the first layer, i.e. ωl = 2 rl/r1 where rl is the resolution of the l-th layer.

Normalization. Since the features in the decoder can have significantly varying statistics from each other, we follow Zhang et al. (2018) and normalize them per channel so that the features in every channel in every layer are zero mean and have unit variance. However, normalizing the feature maps corresponding to the original and denoised latents with different statistics can induce nonzero gradients even when the absolute value has been correctly predicted. To obtain a coherent normalization, we use the feature statistics from the denoised latents to normalize both tensors.

Published as a conference paper at ICLR 2025

3.3 GENERALIZATION FOR LATENT GENERATIVE MODELING

While the bulk of our experiments have been conducted on models trained under DDPM (Ho et al., 2020) for noise prediction, we can generalize our method to different frameworks such as diffusion with velocity prediction (Salimans & Ho, 2022) and flow matching (Lipman et al., 2023). To do this, the only requirement is to be able to estimate the original latents from the model predictions. Under general frameworks such as DDPM and flows, we can write the forward equation in the form t, xt = αtx0 + σtϵt, where αt and σt are increasing (resp. decreasing) functions of t. In Table 1, we provide a summary for these different formulations.

4 EXPERIMENTAL EVALUATION

In this section, we first present our experimental setup, and then go on to present our main results, as well as qualitative results and a number of ablation studies.

4.1 EXPERIMENTAL SETUP

Datasets. We conduct an extensive evaluation on three datasets of different scales and distributions: Image Net-1k (Deng et al., 2009), CC12M (Changpinyo et al., 2021), and S320M: a large internal dataset of 320M stock images. We note that for both Image Net-1k and CC12M, human faces were blurred to avoid training models on identifiable personal data. For both CC12M and S320M, we recaption the images using Florence-2 (Xiao et al., 2024) to obtain captions that more accurately describe the image content. For each of these datasets, we conduct evaluations at both 256 256 and 512 512 image resolution.

Architectures. All experiments are performed using the Multi-modal Di T architecture from Esser et al. (2024). We downscale the model size to be similar to Pixart-α (Chen et al., 2024) and Di TXL/2 (Peebles & Xie, 2023), which corresponds to 28 blocks with a hidden size of 1,536, amounting to a total of 796M parameters. For Image Net-1k models we condition on class labels, while for the other datasets we condition on text prompts. For our main results, we perform our experiments using the asymetric autoencoder from Zhu et al. (2023). For ablation studies, we revert to the lighter autoencoder from SDXL (Podell et al., 2024).

Training and sampling. Unless specified otherwise, we follow the DDPM-ϵ training paradigm (Ho et al., 2020), using the DDIM (Song et al., 2021) algorithm with 50 steps for sampling and a classifier-free guidance scale of λ = 2.0 (Ho & Salimans, 2021). Following Podell et al. (2024), we use a quadratic scheduler with βstart = 0.00085 and βend = 0.012. For the flow experiments, we use the conditional OT probability path (Lipman et al., 2023) with the mode sampling with heavy tails paradigm from Esser et al. (2024). Under this paradigm, the model is trained for velocity prediction and evaluated using the Euler ODE solver. For all velocity models, zero terminal SNR schedule is enforced following Lin et al. (2024).

Similar to Chen et al. (2024), we pre-train all models at 256 resolution on the dataset of interest for 600k iterations. We then enter a second phase of training, in which we optionally apply our perceptual loss, which lasts for 200k iterations for 256 resolution models and for 120k iterations for models at 512 resolution.

When changing the resolution of the images, the resolution of the latents changes by the same factor, keeping the same noise threshold τσ yields inconsistent results across resolutions. To ensure consistent behavior, we follow Esser et al. (2024), and scale the noise threshold similarly to how the noise schedule is scaled in order to keep the same uncertainty per patch. In practice, this amounts to scaling the threshold by the upscaling factor. The kernel sizes for the morphological operations in the outlier detection algorithm are also scaled to cover the same proportion of the image.

Metrics. To evaluate our models, we report results in terms of FID (Heusel et al., 2017) to assess image quality and to what extent the generated images match the distribution of training images, and CLIPScore (Hessel et al., 2021) to assess the alignment between the prompt and the generated image for text-conditioned models. In addition, we report distributional metrics precision and recall (Sajjadi et al., 2018) as well as density and coverage (Naeem et al., 2020) to better understand effects on image quality (precision/density) and diversity (recall/coverage). We evaluate metrics with respect to Image Net-1k and, for models trained on CC12M and S320M, the validation set of CC12M.

Published as a conference paper at ICLR 2025

Goat on the green grass in top of mountain with blurred background.

series takes place within the arena. Mother of the monument on a cloudy day. The top of the building behind a hill. These orange blueberry muffins taste like fluffy little bites of sunshine.

Model in a car showroom.

Figure 3: Samples from models trained with and without our latent perceptual loss on S320M. Samples from the model with perceptual loss (bottom row) show more realistic textures and details.

Pre-training Post-training Results

Res. Iters Res. Iters LPL FID ( ) CLIP ( )

Image Net-1k 256 600k 256 200k 2.98 2.72

512 120k 4.88 3.79

CC12M 256 600k 256 200k 7.81 25.06 6.22 25.12

512 120k 8.79 24.88 7.27 25.12

S320M 256 600k 512 120k 8.81 24.39 8.30 24.41

Table 2: Impact of our perceptual loss for models trained on different datasets and resolutions for DDPM-ϵ model. All models use the same Image Net-256 pretraining for 600k iterations before performing coparing the effect of LPL during post-training. Using LPL boosts FID and CLIP score for all datasets and resolutions considered.

For FID and other distributional metrics, we use the evaluation datasets as the reference datasets and compare an equal number of synthetic samples. For CLIPScore, we use the prompts of the evaluation datasets and the corresponding synthetic samples. Following previous works (Rombach et al., 2022; Peebles & Xie, 2023), we use a guidance scale of 1.5 for resolutions of 256 and 2.0 for resolutions of 512, which we also found to be optimal for our baseline models trained without LPL.

4.2 MAIN RESULTS

LPL applied across different datasets. In Table 2 we consider the impact of the LPL on the FID and CLIPScore for models trained on the three datasets and two resolutions. We observe that the LPL loss consistently improves both metrics across all three datasets. Most notably, FID is improved by 0.91 points on Image Net-1k at 512 resolution and by 1.52 points on CC12M at 512 resolution. The CLIPScore is also improved for both resolutions on CC12M, by 0.06 and 0.24 points respectively. Similarly, for the S320M dataset, we observe that FID is improved by 0.51 points while CLIP score improves (marginally) by 0.02 points. Samples of models trained with and without LPL on CC12M and S320M are shown in Figure 1 and Figure 3, respectively.

Generalization to other frameworks. We showcase the generality of the LPL by applying it to different generative models, experimenting with DDPM for both epsilon and velocity prediction, and flow matching with optimal-transport (OT) path similar to Esser et al. (2024). In Table 3 we report experimental results for models trained on Image Net-1k at 512 resolution. The DDPM-based models perform very similar (except perhaps for FID, where they differ by 0.16 points), and we we find significant improvements across all metrics other than density when using LPL. Density remains similar to the baseline for DDPM models, but improves from 1.14 to 1.29 for the Flow-OT model, where all metrics are improved relative to the DDPM trained ones. We posit that this is due to the mode sampling scheme in (Esser et al., 2024), which emphasizes middle timesteps that could better control the trajectories of the flow path towards having more diversity and not improving the quality (precision/density). Hence, applying LPL to Flow-OT solve this by considerably boosting

Published as a conference paper at ICLR 2025

Paradigm DDPM-ϵ DDPM-vΘ Flow-OT LPL

FID ( ) 4.88 3.79 4.72 3.84 4.54 3.61 Coverage ( ) 0.80 0.82 0.80 0.83 0.82 0.85 Density ( ) 1.14 1.13 1.15 1.14 1.14 1.29 Precision ( ) 0.74 0.77 0.73 0.78 0.75 0.79 Recall ( ) 0.49 0.51 0.49 0.50 0.52 0.54

Table 3: Effect of LPL on Image Net-1k models at 512 resolution trained with different methods. We observe consistent improvements on all metrics when incorporating the LPL, except for density metric for which we observe a very slight degradation when using DDPM training.

(a) F(DLPL) F(Dbase)

(b) F(Dbase) F(Dreal)

(c) F(DLPL) F(Dreal)

Figure 4: Power spectrum of real and generated images. Difference in (log) power spectrum between image generated with and without LPL. Using LPL strenghtens frequencies at the extremes (very low and very high). quality. Notably, considering DDPM baselines, LPL provides a boost as significant as the one provided by using flow matching (scores of DDPM w/ LPL in 2nd and 4th columns are on par or better than Flow-OT w/o LPL in 5th column). Moreover, the provided boost is orthogonal to the training paradigm, leading to overall best results when using LPL with the flow model.

Frequency analysis. While the metrics above provide useful information on model performance, they do not specifically provide insights in terms of frequencies at which using LPL is more effective at modeling data than the baseline. To provide insight on the effect of our perceptual loss w.r.t. the frequency content of the generated images, we compare the power spectrum profile of images generated with a model trained with and without LPL on CC12M at 512 resolution as well as a set of real images from the validation set.

0 50 100 150 200 250 300 350

Frequency f

w. LPL w/o. LPL

Figure 5: Frequency comparison. We compare the power spectrum of the images obtained with or without LPL with real reference images from the validation set of CC12M.

In Figure 4, we plot the difference between log-power spectra between the three image sets. The left-most panel clearly shows the presence of more high frequency signal in the generated images when using LPL to train the model, confirming what has been observed in the qualitative examples of Figure 1 and Figure 3. Moreover, the very lowest frequencies are also strengthened in the samples of the model with LPL. We posit that using the LPL makes it easier to match very low frequencies as they tend be encoded separately in certain channels of the decoder. In Figure 5, we report the error when comparing the power-spectrum of synthetic images and real images, averaged across the validation set of CC12M. For this, we compute the average of the power spectrum across a set of 10k synthetic images from each model and the reference images for the validation set of CC12M. Our experiments indicate that the model trained with LPL is consistently more accurate in modeling high frequencies (f > 150), at the expense of a somewhat larger error at middle frequencies (75 < f < 150).

4.3 ABLATION RESULTS

LPL depth. Using decoder layers to compute our perceptual loss comes with increased computational and memory costs. We therefore study the effect of computing our perceptual loss using only a subset of the decoder layers, as well as a baseline using the RGB pixel output of the decoder. We progressively add more decoder features, so the model with five blocks contains features from the first up to the fifth block. From the results in Figure 7, we find that earlier blocks do not significantly

Published as a conference paper at ICLR 2025

Figure 6: Ablation study on the impact of the noise threshold τσ. We report FID, coverage, density, precision and recall. The dashed line corresponds to the baseline without LPL, note the logarithmic scaling of the noise threshold on the horizontal axis.

0 1 2 3 4 5 RGB Blocks used

3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5

Figure 7: Exploration of LPL depth. Influence of decoder blocks used in LPL on FID, 0 corresponds to not using LPL. Disk radius shows GPU memory usage: w/o LPL=64.9 GB, LPL5 blocks=83.4 GB.

5 10 25 50 100 250 NFE

Baseline w. LPL

Figure 8: Impact of LPL for different number of sampling steps. With higher numbers of sampling steps, the difference between the baseline and the model trained with LPL increases.

0 2 4 6 8 10 w LPL

Figure 9: Influence of the LPL loss weight on model performance. The curve shows a sharp decrease in FID before going back up for larger weights.

improve FID - and can even negatively impact performance. Deeper layers, on the other hand, significantly improve the performance. The most significant gains are obtained when incorporating the third and fourth decoder layers which both improve the FID by more than one point w.r.t. the model incorporating one block less. Finally, the last decoder layer improves the FID only marginally and could be omitted to reduce resource consumption. We perform an additional ablation where the loss operates directly on the RGB image space (without using internal decoder features), which results in degraded performance compared to the baseline while inducing a considerable memory overhead.

Feature normalization. Before computing our perceptual loss, we normalize the decoder features. We compared normalizing the features of the original latent and the predicted one separately, or normalizing both using the statistics from the predicted latent. Our experiment is conducted on Image Net-1k at 512 resolution. While the model trained with separately normalized latents results in a slight boost of FID (4.79 vs. 4.88 for the baseline w/o LPL), the model trained with shared normalization statistics leads to a much more significant improvement and obtains an FID of 3.79.

SNR threshold value. We conduct an experiment on the influence of the SNR threshold which determines at which time steps our perceptual loss is used for training. Lower threshold values correspond to using LPL for fewer iterations that are closest to the noise-free targets. We report results across several metrics in Figure 6 and illustrated with qualitative examples in the supplementary material in Figure 20. We find improved performance over the baseline without LPL for all metrics and that the best values for each metric are obtained for a threshold between three and six, except for the recall which is very stable (and better than the baseline) for all threshold values under 20.

Reweighting strategy. We compare the performance when using uniform or depth-specific weights to combine the contributions from different decoder layers in the LPL. We find that using depthspecific weights results in significant improvements in terms of image quality w.r.t. using uniform weights. While the depth-specific weights achieve an FID of 3.79, the FID obtained using uniform weights is 4.38. Hence, while both strategies improve image quality over the baseline (which achieves an FID of 4.88), reweighting the layer contributions to be approximately similar further boosts performance and improves FID by 0.59 points.

LPL and convergence. As the LPL loss adds a non-negligible memory overhead, by having to backpropagate through the latent decoder, it is interesting to explore at which point in training it should be introduced. We train models on Image Net-1k at 512 resolution with different durations of the post-training stage. We use an initial post-training phase of zero, 50k, or 400k iterations in which LPL is not used, followed by another 120k iterations in which we either apply LPL or not.

Published as a conference paper at ICLR 2025

Initial post-train iters 0 50k 400k

FID ( ) 0.58 0.78 0.97 coverage ( ) +4.29 +3.51 +3.99 density ( ) +0.14 +0.12 +0.21 precision ( ) +4.01 +4.55 +5.89 recall ( ) +1.99 +2.32 +4.22

Table 4: Effect of post-training with LPL. In each column, we report the difference in metrics after post-training for 120k iterations with or without LPL. All metrics improve when adding LPL in the post-training phase.

The results in Table 4 indicate that in each case LPL improves all metrics and that the improvements are larger when the model has been trained longer and is closer to convergence (except for the coverage metric where we see the largest improvement when post-training for only 120k iterations). This suggests that better models (ones trained for longer) benefit more from our perceptual loss.

Influence on sampling efficiency. We conduct an experiment to assess the influence of the perceptual loss on the sampling efficiency. To this end, we sample the Image Net@512 model with different numbers of function evaluations (NFE) then check the trends for the baseline and the model trained with our method. For this experiment, we use DDIM algorithm. Results are reported on Figure 8, where we find that for very low numbers of function evaluations, both models perform similarly. The improvement gains from the LPL loss start becoming considerable after 25 NFEs, where we observe a steady increase in performance gains with respect to the number of function evaluations up to 100, afterwards both models stabilize at a point where the model trained using LPL achieves an improvement of approximately 1.1 points over the baseline.

w/o. LPL w. LPL

Figure 10: Impact of EMA decay rate. Training with LPL is more stable, and allows for a smaller decay parameter.

Impact on EMA. Since the LPL has the effect of increasing the accuracy of the estimated latent during every timestep, it reduces fluctuations between successive iterations of the model during training. Consequently, when training with LPL the EMA momentum can be reduced to obtain optimal performance. In Figure 10 we report the results of a grid search over the momentum parameter γEMA. We find that the model trained with LPL achieves better results when using a slightly lower momentum than the baseline. From the graph, it s clear that better FID is obtained closer to γEMA = 0.99975 for the LPL model, which corresponds to a half-life of approximately 2750 iterations, while the non-LPL model achieves its optimal score at γEMA = 0.9999 corresponding to a half life of 6930 iterations, more than twice as much as the LPL model, thereby validating our hypothesis.

Relative weight. We conduct a grid search over different values for the weight of the LPL loss w LPL. We report FID after training for 120k iterations at 512 resolution, all models are initialized from the same 256 pretrained checkpoint. Our results are reported in Figure 9. Introducing LPL sharply decreases FID for lower weights before going back up at higher weights. We find the model to achieve the best FID for w LP L 3.0 which roughly corresponds to a fifth of the relative contribution to the total loss.

5 CONCLUSION

In this work, we identified a disconnect between the decoder and the training of latent diffusion models, where the diffusion model loss does not receive any feedback from the decoder resulting in perceptually non-optimal generations that oftentimes lack high frequency details. To alleviate this disconnect we introduced a latent perceptual loss (LPL) that provides perceptual feedback from the autoencoder s decoder when training the generative model. Our quantitative results showed that the LPL is generalizable and improves performance for models trained on a variety of datasets, image resolutions, as well as generative model formulations. We observe that our loss leads to improvements from 6% up to 20% in terms of FID. Our qualitative analysis show that the introduction of LPL leads to models that produce images with better structural consistency and sharper details compared to the baseline training. Given its generality, we hope that our work will play an important role in improving the quality of future latent generative models.

Published as a conference paper at ICLR 2025

Dan Amir and Yair Weiss. Understanding and simplifying perceptual distances. In CVPR, 2021.

Jie An, Zhengyuan Yang, Jianfeng Wang, Linjie Li, Zicheng Liu, Lijuan Wang, and Jiebo Luo. Bring metric functions into diffusion models. In IJCAI, 2024.

Pietro Astolfi, Marlene Careil, Melissa Hall, Oscar Mañas, Matthew Muckley, Jakob Verbeek, Adriana Romero Soriano, and Michal Drozdzal. Consistency-diversity-realism Pareto fronts of conditional image generative models. ar Xiv preprint, 2406.10429, 2024.

Tariq Berrada, Jakob Verbeek, Camille Couprie, and Karteek Alahari. Unlocking pre-trained image backbones for semantic image synthesis. In CVPR, 2024.

Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR, 2019.

Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing webscale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021.

Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-α: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In ICLR, 2024.

Ricky T. Q. Chen. torchdiffeq, 2018. URL https://github.com/rtqichen/ torchdiffeq.

Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. In CVPR, 2022.

Steffen Czolbe, Oswin Krause, Ingemar J. Cox, and Christian Igel. A loss function for generative neural networks based on Watson s perceptual model. In Neur IPS, 2020.

Xiaoliang Dai, Ji Hou, Chih-Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, Matthew Yu, Abhishek Kadian, Filip Radenovic, Dhruv Mahajan, Kunpeng Li, Yue Zhao, Vladan Petrovic, Mitesh Kumar Singh, Simran Motwani, Yi Wen, Yiwen Song, Roshan Sumbaly, Vignesh Ramanathan, Zijian He, Peter Vajda, and Devi Parikh. Emu: Enhancing image generation models using photogenic needles in a haystack. ar Xiv preprint, 2309.15807, 2023.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Image Net: A large-scale hierarchical image database. In CVPR, 2009.

Prafulla Dhariwal and Alex Nichol. Diffusion models beat GANs on image synthesis. In Neur IPS, 2021.

Patrick Esser, Robin Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.

Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In ICML, 2024.

Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer is a strong image synthesizer. In ICCV, 2023.

Tiankai Hang and Shuyang Gu. Improved noise schedule for diffusion training. ar Xiv preprint, 2407.03297, 2024.

Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via min-snr weighting strategy. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7441 7451, October 2023a.

Published as a conference paper at ICLR 2025

Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via Min-SNR weighting strategy. In ICCV, 2023b.

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation metric for image captioning. In EMNLP, 2021.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Neur IPS, 2017.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In Neur IPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Neur IPS, 2020.

Jonathan Ho, Chitwan Saharia, William Chan, David J. Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47), 2022a.

Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. In Neur IPS, 2022b.

Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In CVPR, 2017.

Younghyun Jo, Sejong Yang, and Seon Joo Kim. Investigating loss functions for extreme superresolution. In Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2020.

Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016.

Minguk Kang, Richard Zhang, Connelly Barnes, Sylvain Paris, Suha Kwak, Jaesik Park, Eli Shechtman, Jun-Yan Zhu, and Taesung Park. Distilling diffusion models into conditional GANs. In ECCV, 2024.

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.

Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of Style GAN. In CVPR, 2020.

Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. In Neur IPS, 2021.

Diederik Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.

Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In Neur IPS, 2019.

Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. Photorealistic single image super-resolution using a generative adversarial network. In CVPR, 2017.

Mark Levy, Bruno Di Giorgi, Floris Weers, Angelos Katharopoulos, and Tom Nickson. Controllable music production with diffusion models and guidance gradients. In Neur IPS Workshop on Diffusion Models, 2023.

Shanchuan Lin and Xiao Yang. Diffusion model with perceptual loss. ar Xiv preprint, 2401.00110, 2024.

Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. In WACV, 2024.

Published as a conference paper at ICLR 2025

Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In ICML, 2023.

Nanye Ma, Mark Goldstein, Michael S. Albergo, Nicholas M. Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers, 2024. URL https://arxiv.org/abs/2401.08740.

Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2022.

Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In ICML, 2020.

Mang Ning, Mingxiao Li, Jianlin Su, Albert Ali Salah, and Itir Onal Ertugrul. Elucidating the exposure bias in diffusion models. In ICLR, 2024.

Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In CVPR, 2019.

William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In ICLR, 2024.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In CVPR, 2022.

Mehdi S. M. Sajjadi, Olivier Bachem, Mario Luˇci c, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. In Neur IPS, 2018.

Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In ICLR, 2022.

Robin San Roman, Yossi Adi, Antoine Deleforge, Romain Serizel, Gabriel Synnaeve, and Alexandre Defossez. From discrete tokens to high-fidelity audio using multi-band diffusion. In Neur IPS, 2023.

Edgar Schönfeld, Vadim Sushko, Dan Zhang, Juergen Gall, Bernt Schiele, and Anna Khoreva. You only need adversarial supervision for semantic image synthesis. In ICLR, 2021.

Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a-video: Text-to-video generation without text-video data. In ICLR, 2023.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.

Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In ICML, 2023.

Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution-robust large mask inpainting with fourier convolutions. In WACV, 2022.

Lokesh Veeramacheneni, Moritz Wolter, Hildegard Kuehne, and Juergen Gall. Fréchet wavelet distance: A domain-agnostic metric for image generation. ar Xiv preprint, 2312.15289, 2023.

Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 23(7):1661 1674, 2011.

Published as a conference paper at ICLR 2025

Tong Wu, Zhihao Fan, Xiao Liu, Yeyun Gong, Yelong Shen, Jian Jiao, Hai-Tao Zheng, Juntao Li, Zhongyu Wei, Jian Guo, Nan Duan, and Weizhu Chen. AR-Diffusion: Auto-regressive diffusion model for text generation. In Neur IPS, 2023.

Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for a variety of vision tasks. In CVPR, 2024.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.

Zixin Zhu, Xuelu Feng, Dongdong Chen, Jianmin Bao, Le Wang, Yinpeng Chen, Lu Yuan, and Gang Hua. Designing a better asymmetric VQGAN for Stable Diffusion. ar Xiv preprint, 2306.04632, 2023.