Know that the account with the same name is released synchronously

Today, I'll read a paper on self supervised image denoising ：

subject ：Blind2Unblind: Self-Supervised Image Denoising with Visible Blind Spots
paper：https://arxiv.org/abs/2203.06967
code：https://github.com/demonsjin/Blind2Unblind

Pre knowledge

Need to know Noise2Void, If you don't understand , You can read what I wrote note .

After watching the , We know , about Noise2void（N2V） Come on , Given a sheet noisy image y, First put some points on it mask fall , Then input it to the neural network , Neural network output target by y In itself , In this way, the network can learn the method of denoising .

The above method is called blindspots schemes, Because it mask It fell off input Some of pixels, Make the Internet invisible pixel.

Blind2Unblind It is mentioned in the article “relief from identity mapping”, It points out the method with blind spots , Can avoid identity mapping . The so-called identity mapping , It means if you are blindspots schemes in , Don't go to mask fall input Some of pixels, But will be complete y Input to neural network , And neural network target It's output y In itself , Then the network will learn identity mapping , Instead of learning how to denoise .

But on the other hand , The author also suggests that ,blindspots schemes There are also shortcomings. , That will lead to information loss, Because the Internet can't see being mask The information of these points .

Innovation points

Author points out ,N2V This kind of method , Due to the existence of blind spots , It can avoid network learning identity mapping , At the same time, according to mathematical analysis and experiments , The network can learn denoising .

So the author is going to use blindspots schemes, But the author also doesn't want to bear information loss, So when designing the architecture , He used raw noisy image y, That is to say, not to carry out mask Operation of the 、 The original noisy image. such , You can use all input Pixel information , Theoretically, it avoids information loss.

Model architecture

a sheet noisy image y Through one global masker Ω operation , Become four pictures with blind spots , These four images are denoised through the network f, Four denoising results are obtained , And then through global mask mapper h The operation of , Get a denoising result $h(f_{\theta }({\Omega }_y))$. On the other hand , complete noisy image y Also through denoising neural network f, Get another denoising result $f_{\theta }(y)$. For the sake of simplicity , The former I call “ Denoising results with blind spots ”, The latter I call “ Denoising results without blind spots ”. A weighted average of the two , The results obtained and y do loss, namely $L_{rev}$,rev It means re-visible, That is, it was originally a blindspots The plan , Now because of the introduction of raw noisy image y, It becomes less blind 了 , No longer need to bear information loss 了 .$L_{reg}$ It's a regularization term , The meaning is self-evident .

In the architecture diagram Ω and h And it's very simple . The author uses 4×4 Give an example of , First divide the picture into several 2×2 Of cell, For each at the same time cell Top left corner of 、 Upper right corner 、 The lower left corner 、 In the lower right corner mask operation , Get four pictures with blind spots , Input denoising network , Four denoising results are obtained . The gray in the four denoising results represents the position corresponding to the blind spot before denoising pixel,h All you have to do is put all the gray pixel, The relative position does not change , Extract it to form a picture , That's the gray picture on the far right , It is the denoising result corresponding to all blind spots . The top half of the image above , The corresponding is the complete y The denoising results obtained （ No blind spots ）. The two denoising results should be used together to train .

inference Stage

Just pay attention to the picture above , It means that the denoising result without blind spots , In terms of experience, we should have better results , therefore inference Phase only uses it .

experiment

The synthetic image denoising is done 、 Real image denoising , In some cases, the index is not the highest , The remarkable experimental results in the excerpt are as follows ：

This is a SIDD The result on , It can be seen that it is better than N2V It is better to , But and NBR2NBR There is no significant difference .