Improving Diffusion Model Efficiency Through Patching

This paper is mainly innovative in thinking. DiffusionRequires hundreds of sampling steps, 这是DiffusionImportant reasons for slow sampling. 因此, a lot of accelerationDiffusionThe method focuses on how to reduce the number of sampling steps. The author of this paper proposes another way of thinking, That is, by dividing the imagepatch降低DiffusionSampling efficiency and memory footprint for each step of the model.

1. Introduction

对于图像数据, A very efficient way to reduce computational cost is to process low-dimensional representations of image data, This low-dimensional representation can be obtained by downsampling or an encoder, But these two methods also need to upsample or decode the low-dimensional data., need more network. This article adopts a more convenient approach, Is divided into imagespatch然后直接concatenate起来.

The main contributions of the authors of this paper:

1. 对DiffusionIn-depth analysis of training objectives, shows that in many sampling steps, High-resolution convolutional layers are redundant.
2. 提出了一种Patched Diffusion Model(PDM), And it is proved that the cost of sampling can be significantly reduced. And this method can be used concurrently with the existing downsampling method.
3. The author's analysis of different training objectivesDiffusioncompare the effects, found whether or notpatch, Models are more robust by predicting data than by predicting noise.
4. 作者提出了一种将PDMfor more complex,Higher resolution method of the data, That is, dividing a network into multiple parts, 然后在256x256的ImageNethigh-quality results.

2. Patch Diffusion Model

DiffusionThe principle of the model will not be repeated, It's just that the markup used by the author here is the same asDDPM不太一样, Put a picture at a glance.

训练的目标, that is to rebuildx

这里的 ${\gamma }_t$ 是常数. But if the target is added during training ${\gamma }_t$ On the contrary, the quality of the sampling will be degraded, 因为会 overemphasize high frequency information. So the author in this paper ${\gamma }_t=\sqrt{\frac{{\alpha }_t}{1-{\alpha }_t}}$, to take into account low-frequency information.

Diffusion的训练过程, 也就是将 x 进行 perturb 得到 $z_t$, 然后训练网络 $x_{\theta }$ 通过 $z_t$ 重建出 x, The training goal of the figure above.

2.1 DiffusionRedundancy in model architecture

作者指出, for any hidden variable $z_t$, may be many different $x$ by adding different noises. 但是这些 $x$ can correspond to this $z_t$, 只是概率不同. 这种情况在 t 比较大, That is, when the noise is relatively loud, it is more obvious. 怎么理解呢?

As shown in the figure, three apples of different colors and shapes are exactly the same, Then, through the analysis of the implicit experimental results above：变量(That is, an image that approximates noise)预测得到的 x could be any of these three apples. 在这种情况下, What is the best result after model optimization??

作者指出, The optimal result obtained by optimizing the above training objective can be expressed as:

推导的过程如下:

简而言之, That is to say, when the model converges to the optimum at each step, 预测得到的 $x^{\ast }$ 是输入的 $z_t$ corresponding to all $x$ 的期望, Or it can also be seen as the mean. Substitute into the apple example just now, The model predicts that the apple will be an average of the three colors. In fact, many usesL2Similar to the case where the loss is optimized, That is, what the model eventually learns is the mean of all possible cases., thereby losingdiversity. Not only is the coloring model tends to predict results as gray, 还体现在DiffusionThe model may predict that high-frequency information will be blurred, because the average.

But the author did not say how to solve the vague problem, but based on this vague fact, Point out that since the direct prediction when the noise is relatively large x 是模糊的, That is, high-frequency information cannot be obtained very well., Then there is no need for such a high computational cost, It's blurry anyway. So there is no need to do convolution at high resolution.

2.2 Patched Diffusion Models(PDM)

First of all, how does the authorPatch操作, 对于(H, W, C)的图像, the author according to P The size is divided into non-overlappingPatch, 然后concatenate起来, So convert the image to $(H/P,W/P,C\times P^2)$ 的数据, 这个作为Unet的输入.

作者尝试了不同的patch大小2,4,8. 发现patch=4的时候, Both have better sampling quality, also increase the speed. And by comparing different authorpatch的RMSEIt is confirmed that when the noise is relatively large, Convolution at high resolution is useless.

After that, the author tried different training objectives, 也就是重建 $ϵ$, $x$, $\sqrt{{\alpha }_t}ϵ-\sqrt{1-{\alpha }_t}x$The impact of three different goals on quality. The conclusion is to rebuild $x$, $\sqrt{{\alpha }_t}ϵ-\sqrt{1-{\alpha }_t}x$ than rebuild $ϵ$ 更鲁棒.

2.3 Scaling to more complex datasets

这个部分很简单, 就是用了两个Unet, 一个不分patchUsed in the first half of the blurring stage, The other is used in the second half.

For more experimental results, please refer to the original text.