赛题背景

Video enhancement and super-resolution areCVone of the core algorithms,It is of great significance to improve the quality and clarity of early film video.This question is for a bunch of videos（低分辨率和高分辨率）,Use the trained modelPredict the low-resolution video to get the high-resolution video.

全代码

导入工具包

import cv2
import numpy as np
import tensorflow as tf
from tensorflow import keras

from tensorflow.keras.layers import Conv2D
from tensorflow.keras.layers import Conv2DTranspose
from tensorflow.keras.layers import InputLayer
from tensorflow.keras.models import Sequential

读取图片

path = "./h_GT/Youku_00000_h_GT/001.bmp"
img_GT = cv2.imread(path)/255.0
img_GT.shape

path = "./l/Youku_00000_l/001.bmp"
img_l = cv2.imread(path)/255.0
img_l.shape

实现FSRCNN网络

def fsrcnn():

    model = Sequential()
    model.add(InputLayer(input_shape=(270, 480, 3)))
    
    # first_part
    model.add(Conv2D(56, 5, padding='same', activation='relu'))
    
    # mid_part
    model.add(Conv2D(12, 1, padding='same', activation='relu'))
    for i in range(4):
        model.add(Conv2D(12, 3, padding='same', activation='relu'))
        
    # last_part
    model.add(Conv2DTranspose(3, 9, strides=4, padding='same',))
    
    model.compile(optimizer=tf.optimizers.Adam(1e-1), loss=tf.losses.mse, metrics=['mse'])
    return model

FSRCNN

FSRCNN (Faster Super-Resolution Convolutional Neural Network) 解决了SRCNN的耗时问题.FSRCNN使用反卷积Upsampling is used to replace the preprocessing of interpolation,End-to-end learning is straightforward;FSRCNN使用ResNet bottleneck architecture to improve model accuracy,Use smaller convolution kernels and more convolutional layers instead of large convolution kernels.So when generating different high-definition resolution pictures,FSRCNNJust adjust the deconvolution weights used for upsampling,The rest of the convolutional layers remain unchanged,Greatly speed up training,even in real time.

FSRCNN模型训练

# 使用模型
model = fsrcnn()

# 模型监控：自动调节学习率
plateau = keras.callbacks.ReduceLROnPlateau(monitor='val_loss', verbose=0, mode='min', factor=0.10, patience=6)
# The model stops when it reaches optimality on the validation set
early_stopping = keras.callbacks.EarlyStopping(monitor='val_loss', verbose=0, mode='min', patience=25)
# The model is kept at the optimum point
checkpoint = keras.callbacks.ModelCheckpoint('fsrcnn.h5', monitor='val_loss', verbose=0, mode='min', save_best_only=True)

# 训练数据
x = np.array([img_l,img_l])
y = np.array([img_GT,img_GT])

# 模型训练
model.fit(x, y, epochs=10, batch_size=2, verbose=1, shuffle=True, validation_data=(x, y), callbacks=[plateau, early_stopping, checkpoint])

FSRCNN模型验证

model.evaluate(x, y, verbose=0)

FSRCNN模型预测

pic_super = model.predict(x, verbose=0, batch_size=1)

Save the picture to view

cv2.imwrite("./fsrcnn_00.bmp", pic_super[0])

ESPCN

ESPCN（Efficient Sub-Pixel Convolutional Neural Network）吸收了FSRCNN的精华,It is only used at the end of the model亚像素卷积way of upsampling,This preserves more texture area in low-res space,It is also possible to do real-time in the video superscore.

实现ESPCN网络

def espcn():
    inputs = keras.layers.Input(shape=(270, 480, 3))
    cnn = keras.layers.Conv2D(64, 5, padding='same', activation='relu')(inputs)
    cnn = keras.layers.Conv2D(32, 3, padding='same', activation='relu')(cnn)
    cnn = keras.layers.Conv2D(3 * 4 **2, 3, padding='same')(cnn)
    cnn = tf.reshape(cnn, [-1, 270, 480, 4, 4, 3])
    cnn = tf.transpose(cnn, perm=[0, 1, 3, 2, 4, 5]) 
    outputs = tf.reshape(cnn, [-1, 270 * 4, 480 * 4, 3])
    
    model = keras.models.Model(inputs=[inputs], outputs=[outputs])
    model.compile(optimizer=tf.optimizers.Adam(1e-1), loss=tf.losses.mse, metrics=['mse'])
    return model

ESPCN模型训练

# 使用模型
model = espcn()

# 模型监控：自动调节学习率
plateau = keras.callbacks.ReduceLROnPlateau(monitor='val_loss', verbose=0, mode='min', factor=0.10, patience=6)
# The model stops when it reaches optimality on the validation set
early_stopping = keras.callbacks.EarlyStopping(monitor='val_loss', verbose=0, mode='min', patience=25)
# The model is kept at the optimum point
checkpoint = keras.callbacks.ModelCheckpoint('espcn.h5', monitor='val_loss', verbose=0, mode='min', save_best_only=True)

# 训练数据
x = np.array([img_l,img_l])
y = np.array([img_GT,img_GT])

# 模型训练
model.fit(x, y, epochs=10, batch_size=2, verbose=1, shuffle=True, validation_data=(x, y), callbacks=[plateau, early_stopping, checkpoint])

ESPCN模型验证

model.evaluate(x, y, verbose=0)

ESPCN模型预测

pic_super = model.predict(x, verbose=0, batch_size=1)

Save the picture to view

cv2.imwrite("./espcn_00.bmp", pic_super[0])

以上内容和代码全部来自于《阿里云天池大赛赛题解析(深度学习篇)》这本好书,十分推荐大家去阅读原书！