@TOC
Deep Learning on Image Denosing: An Overview
Abstract
基于深度学习的判别学习对于解决高斯噪声的问题十分游泳，并且对于真实的噪声也有一定的效果
本文分为几部分：白噪声图像；真实噪声图像；盲去噪；包含噪声、模糊和低分辩率的混合图像。分析了不同深度学习方法的动机和准则，与最新的方法作对比，最后指出了潜在的挑战和未来研究方向
Q：白噪声？盲去噪？
1 Introduction
相机捕捉的是已经退化的图像，受到光和噪声的干扰，噪声在传输和潜在压缩中产生
50年前非线性和非自适应滤波器被用于图像去噪，之后基于稀疏的方法被应用，为了恢复潜在图像的细节，先验知识（总变分正则）可以使得噪声图像变得光滑，虽然很多方法效果不错，但是都有各自的缺点，例如有的需要对测试集进行优化
深度学习的技术发展开来，并逐渐被改进，例如损失函数MSE到entropy损失，对偶拉格朗日乘子，贪婪算法等被用于神经网络，之后构建一个全新的神经网络结构对于去噪很有意义，不论是增强他的深度或者改变他的激活函数
卷积神经网络被提出，但是面临很多的问题，例如梯度消失，激活函数的计算损失较大，硬件设备的计算速度较慢。2012年，Alex网络被提出，在2015年，深度神经网络被应用于图像去噪，DnCNN解决多重low-level任务，包含conv、batchnorm和Relu。Resnet被提出解决去噪、超分辨和JPEG
去遮挡。为了权衡去噪的表现和速度，CNLNet被提出来有效的取出彩图噪声
FFDNet提升了去噪速度并且可以处理盲噪声。GAN的提出可以处理非配对的噪声图像
2 Fundamental frameworks of deep learning methods for image denoising
2.1 Machine learning methods for image denoising
监督学习：
无监督学习：给定训练样本寻找pattern而不是标签匹配
半监督学习：将模型应用于给定的数据分布来学习，然后用来给无标签样本标签
2.2 Neural networks for image denoising
神经网络包括神经元，输入，激活函数
，权重
和偏置
多层神经网络被称为MLP，下图中包含两层，隐藏层和输出层
process
2.3 CNNs for image denoising
LeNet—AlexNet—VGGNet—GoogLeNet
VGG
GoogLeNet
Drawbacks：网络很深可能会产生梯度消失或者梯度爆炸；过拟合——>解决ResNet
ResNet
GAN
Generator and Discriminator
2.4 Hardware and software used in deep learning
Hardware : Nvidia AMD
Software : Caffe Theano Matconvnet TensorFlow Keras PyTorch
3 Deep learning techniques in image denoising
3.1 Deep learning techniques for additive white noisy-image denoising
白噪声图片（AWNIs）
Gaussian，Poisson，Salt，Pepper，Multiplicative noisy images
3.1.1 CNN/NN for AWNI denoising
2017 DnCNN
多尺度底层视觉工作：图像去噪，超分辨，去块滤波
残差学习——深层神经网络——递归和自适应特征提取——改进CNNs，深度神经网络，残差学习，多尺度知识（计算复杂度较大，内存损耗多）——增强感受野——通过多尺度交叉路径的残差连接压制噪声
改变网络结构框架
-
从CNN的多个输入中融合特征：一个样本的不同部分作为不同网络的多个输入；一个样本作为输入的不同视角；不同通道作为CNN的输入
-
改变损失函数
-
增强CNN的深度或宽度：通过增强网络的深度和宽度来扩大感受野的大小
-
增加CNN的一些辅助插件：例如激活函数，空洞卷积（dilated convolution），全连接层和池化操作来增强CNN的表达能力
-
引入跳跃连接或级联操作：跳跃连接能增强级联操作
3.1.2 CNN/NN and common feature extraction methods for AWNI denoising
Weak edge-information noisy images
进行域与域之间的转变
non-linear noisy images
CNN提取特征；利用核函数的方法将获取的非线性特征映射到线性；利用残差学习重建潜在清晰图像
high dimensional noisy images
CNN与降维方法相结合。PCA：卷积操作提取特征，PCA降维，卷积处理降维的特征重建图像
non-salient noisy images
信号处理可以让CNN提取非显著特征，例如，跳跃连接
tasks involving high computational costs
3.1.3 Combination of optimization method and CNN/NN for AWNI denoising
discriminative learning；prior knowledge via regular term of loss function
denosing speed
GAN估计噪声，处理其他任务，图像恢复和超分辨率
基于CNN经验的贪婪算法和迁移学习能加速普遍的算法
denoising performance
基于CNN的优化方法使得噪声图片更加光滑，CNN的全变分去噪减少了噪声像素的有效性。GAN最近邻算法有效地噪声
3.2 Deep learning techniques for real noisy image denoising
single end-to-end CNN
改变网络结构。多尺度知识对图像去噪很有效。
提取细节特征：recurrent connections
处理未知真实噪声：residual structure was utilized to facilitate low-frequency features
提取通道更多潜在特征：attention
the combination of prior knowledge and CNN
3.3 Deep learning techniques for blind denoising
3.4 Deep learning techniques for hybrid noisy image denoising
混合噪声图像：噪声、模糊、低分辨率等
突发技术burst techniques：suffered from effects of noise and camera shake
4 Experimental results
4.1 Datasets
4.1.1 Training datasets
灰度噪声gray-noisy
训练高斯去噪和盲去噪：
- BSD400 dataset：
png：400；size：180*180 - Waterloo Exploration Database：
png：4744 nature images
颜色噪声color-noisy
- BSD432
- Waterloo Exploration Database
- polyU-Real-World-Noisy-Images datasets：
real noisy images：100；size：2784*1856
4.1.2 Test datasets
灰度噪声gray-noisy
测试高斯噪声和盲噪声：
- Set12：contain 12 scenes
- BSD68：contain 68 nature images
颜色噪声color-noisy
- CBSD68
- Kodak24：24 color noisy images
- McMaster：18 color noisy images
- cc：15 real noisy images of different ISO(1600, 3200, 6400)
- DND：50 real noisy images(clean images were captured by low-ISO images)
- NC12：12 noisy images(didn’t have ground-truth clean images)
- SIDD：320 images pairs of noisy and ground-truth images(real noisy images from smart phones)
- Nam：11 scenes(saved in JPGE format)
4.2 Experimental resultes
4.2.1 Deep learning techniques for additive white noisy-image denoising
Table10：灰度图附加白噪声
Table11：FSIM
放大图片的局部区域，观察去噪效果
4.2.2 Deep learning techniques for real-noisy image denoising
Table15 16：DRDN has the best performance
Table 17：compressed noisy images. AGAN obtains excellent performance.
Table18：real noisy images of different IOS value, SDNet and BRDNet achieved the best and the second-best denoising performance.
4.2.3 Deep learning techniques for blind denoising
blind denoising: noise which is complex in the real world and not subject to rules
FFDNet and ADNet are superior to other methods in blind denoising
4.2.4 Deep learning techniques for hybrid-noisy-image denoising
deep learning techniques based multi-degradation idea：WarpNet DnCNN and MemNet
5 Discussion
Improving denoising performance kills
- Enlarge the receptive field can capture more context information: increasing the depth and width of the networks. However, this results in higher computational costs and more memory consumption——>solve: dilated convolution——>higher performance and efficiency, edge information
- extra information(prior knowledge) combined with CNN: loss function
- combine local and global information: enhance the memory abilities of the shallow layers on deep layers to better filter the noise——>operation: residual operation and recursive operation
- single processing methods: ?
- Data augmentation: horizontal flip, vertical flip and color jittering; Using GAN to construct virtual noisy images is also useful for image denoising
- Transfer learning, graph and neural architecture search
- imporve hardware or camera mechanism
trick:
- compress neural networks,
- reduce the depth or the width of deep neural networks: reduce complexity of these networks
- use small convolutional kernal and group convolution: reduce the number of parameters, accelerating the speed of training
- fusion of dimension reduction methods: PCA and CNN
Complex noisy images: step by step processing, for example, two steps: one recover high resolution image by CNN, another uses a novel CNN to filter the noise of high-resolution image
several as yet unsolved problems
- deeper denoising networkds require more memory resources
- deeper denoising networks is not a stable solution for real noisy image, unpaired noisy image and muliti-degradation tasks
- real noisy images are not easily captured
- difficult to solve unsupervised denoising tasks
- more accurate metrics need to be found for image denoising
6 Conclusion
regular Gaussian noise has achieved great success, while the noise in the real world is complex and irregular
Imporve hardware devices in order to suppress the noise for capturing a high-quality image
noisy images with ground truth