Torch implementation of "Enhanced Deep Residual Networks for Single Image Super-Resolution"

Overview

NTIRE2017 Super-resolution Challenge: SNU_CVLab

Introduction

This is our project repository for CVPR 2017 Workshop (2nd NTIRE).

We, Team SNU_CVLab, (Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee of Computer Vision Lab, Seoul National University) are winners of NTIRE2017 Challenge on Single Image Super-Resolution.

Our paper was published in CVPR 2017 workshop (2nd NTIRE), and won the Best Paper Award of the workshop challenge track.

Please refer to our paper for details.

If you find our work useful in your research or publication, please cite our work:

[1] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee, "Enhanced Deep Residual Networks for Single Image Super-Resolution," 2nd NTIRE: New Trends in Image Restoration and Enhancement workshop and challenge on image super-resolution in conjunction with CVPR 2017. [PDF] [arXiv] [Slide]

@InProceedings{Lim_2017_CVPR_Workshops,
  author = {Lim, Bee and Son, Sanghyun and Kim, Heewon and Nah, Seungjun and Lee, Kyoung Mu},
  title = {Enhanced Deep Residual Networks for Single Image Super-Resolution},
  booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
  month = {July},
  year = {2017}
}

In this repository, we provide

  • Our model architecture description (EDSR, MDSR)
  • NTIRE2017 Super-resolution Challenge Results
  • Demo & Training code
  • Trained models (EDSR, MDSR)
  • Datasets we used (DIV2K, Flickr2K)
  • Super-resolution examples

The code is based on Facebook's Torch implementation of ResNet (facebook/fb.resnet.torch).

We also provide PyTorch version of EDSR and MDSR. (Until now, only some models are available.)

Model Architecture

EDSR (Single-scale model. We provide scale x2, x3, x4 models).

EDSR

MDSR (Multi-scale model. It can handle x2, x3, x4 super-resolution in a single model).

MDSR

Note that the MDSR architecture for the challenge and for the paper[1] is slightly different. During the challenge, MDSR had variation between two challenge tracks. While we had scale-specific feature extraction modules for track 2:unknown downscaling, we didn't use the scale-specific modules for track 1:bicubic downscaling.

We later unified the MDSR model in our paper[1] by including scale-specific modules for both cases. From now on, unless specified as "challenge", we describe the models described in the paper.

NTIRE2017 Super-resolution Challenge Results

We proposed 2 methods and they won the 1st (EDSR) and 2nd (MDSR) place.

Challenge_result

We have also compared the super-resolution performance of our models with previous state-of-the-art methods.

Paper_result

About our code

Dependencies

  • Torch7
  • cuDNN
  • nccl (Optional, for faster GPU communication)

Our code is tested under Ubuntu 14.04 and 16.04 environment with Titan X GPUs (12GB VRAM).

Code

Clone this repository into any place you want. You may follow the example below.

makeReposit = [/the/directory/as/you/wish]
mkdir -p $makeReposit/; cd $makeReposit/
git clone https://github.com/LimBee/NTIRE2017.git

Quick Start (Demo)

You can test our super-resolution algorithm with your own images.

We assume the images are downsampled by bicubic interpolation.

Model Scale File Name Self Esemble # ResBlocks # Filters # Parameters
EDSR baseline x2 baseline_x2.t7 X 16 64 1.5M
EDSR baseline x3 baseline_x3.t7 X 16 64 1.5M
EDSR baseline x4 baseline_x4.t7 X 16 64 1.5M
MDSR baseline Multi baseline_multiscale.t7 X 16 64 3.2M
EDSR x2 EDSR_x2.t7 X 32 256 43M
EDSR x3 EDSR_x3.t7 X 32 256 43M
EDSR x4 EDSR_x4.t7 X 32 256 43M
MDSR Multi MDSR.t7 X 80 64 8.0M
EDSR+ x2 EDSR_x2.t7 O 32 256 43M
EDSR+ x3 EDSR_x3.t7 O 32 256 43M
EDSR+ x4 EDSR_x4.t7 O 32 256 43M
MDSR+ Multi MDSR.t7 O 80 64 8.0M

  1. Download our models

    cd $makeReposit/NTIRE2017/demo/model/
    
    # Our models for the paper[1]
    wget https://cv.snu.ac.kr/research/EDSR/model_paper.tar

    Or, use the link: model_paper.tar

    (If you would like to run the models we used during the challenge, please contact us.)

    After downloading the .tar files, make sure that the model files are placed in proper locations. For example,

    $makeReposit/NTIRE2017/demo/model/bicubic_x2.t7
    $makeReposit/NTIRE2017/demo/model/bicubic_x3.t7
    ...
  2. Place your low-resolution test images at

    $makeReposit/NTIRE2017/demo/img_input/

    The demo code will read .jpg, .jpeg, .png format images.

  3. Run test.lua

    You can run different models and scales by changing input arguments.

    # To run for scale 2, 3, or 4, set -scale as 2, 3, or 4
    # To run EDSR+ and MDSR+, you need to set -selfEnsemble as true
    
    cd $makeReposit/NTIRE2017/demo
    
    # Test EDSR (scale 2)
    th test.lua -model EDSR_x2 -selfEnsemble false
    
    # Test EDSR+ (scale 2)
    th test.lua -model EDSR_x2 -selfEnsemble true
    
    # Test MDSR (scale 2)
    th test.lua -model MDSR -scale 2 -selfEnsemble false
    
    # Test MDSR+ (scale 2)
    th test.lua -model MDSR -scale 2 -selfEnsemble true

    (Note: To run the MDSR, model name should include multiscale or MDSR. e.g. multiscale_blahblahblah.t7)

    The result images will be located at

    $makeReposit/NTIRE2017/demo/img_output/
    • Here are some optional argument examples you can adjust. Please refer to the following explanation.
    # You can test our model with multiple GPU. (n = 1, 2, 4)
    -nGPU       [n]
    
    # You must specify this directory. Default is /var/tmp/dataset
    -dataDir    [$makeData]
    -dataset    [DIV2K | myData]
    -save       [Folder name]
    
    # Please see our paper[1] if you want to know about self-ensemble.
    -selfEnsemble   [true | false]
    
    # Please reduce the chopSize when you see 'out of memory'.
    # The optimal size of S can be vary depend on your maximum GPU memory.
    -chopSize   [S]   
  4. (Optional) Evaluate PSNR and SSIM if you have ground-truth HR images

    Place the GT images at

    $makeReposit/NTIRE2017/demo/img_target

    Evaluation is done by running the MATLAB script.

    matlab -nodisplay <evaluation.m

    If you don't want to calculate SSIM, please modify evaluation.m file as below. (Calculating SSIM of large image is very slow for 3 channel images.)

    line 6:     psnrOnly = false; -> psnrOnly = true;
    

You can reproduce our final results by running makeFinal.sh in NTIRE2017/demo directory. Please uncomment the command you want to execute in the file.

sh makeFinal.sh

Dataset

If you want to train or evaluate our models with DIV2K or Flickr2K dataset, please download the dataset from here. Place the tar file to the location you want. (We recommend /var/tmp/dataset/) If the dataset is located otherwise, you have to change the optional argument -dataset for training and test.

  • DIV2K from NTIRE2017

    makeData = /var/tmp/dataset/ # We recommend this path, but you can freely change it.
    mkdir -p $makeData/; cd $makedata/
    tar -xvf DIV2K.tar

    You should have the following directory structure:

    /var/tmp/dataset/DIV2K/DIV2K_train_HR/0???.png
    /var/tmp/dataset/DIV2K/DIV2K_train_LR_bicubic/X?/0???.png
    /var/tmp/dataset/DIV2K/DIV2K_train_LR_unknown/X?/0???.png

  • Flickr2K dataset collected by ourselves using Flickr API

    makeData = /var/tmp/dataset/
    mkdir -p $makeData/; cd $makedata/
    wget https://cv.snu.ac.kr/research/EDSR/Flickr2K.tar
    tar -xvf Flickr2K.tar

    You should have the following directory structure:

    /var/tmp/dataset/Flickr2K/Flickr2K_HR/00????.png
    /var/tmp/dataset/Flickr2K/Flickr2K_train_LR_bicubic/X?/00????x?.png
    /var/tmp/dataset/Flickr2K/Flickr2K_train_LR_unknown/X?/00????x?.png

    We also provide the code we used for collecting the Flickr2K images at

    $makeReposit/NTIRE2017/code/tools/Flickr2K/

    Use your own flickr API keys to use the script.

    During the challenge, we additionally generated training data by learning simple downsampler networks from DIV2K dataset track 2.
    You can download the downsampler models from here.

To make data loading faster, you can convert the dataset into binary .t7 files

  • Convert DIV2K dataset from .png to into .t7 files
    cd $makeReposit/NTIRE2017/code/tools
    
    # Choose one among below
    
    # This command generates multiple t7 files for
    # each image in DIV2K_train_HR folder (Requires ~2GB RAM for training)
    th png_to_t7.lua -apath $makeData -dataset DIV2K -split true
    
    # This command generates a single t7 file that contains
    # every image in DIV2K_train_HR folder (Requires ~16GB RAM for training)
    th png_to_t7.lua -apath $makeData -dataset DIV2K -split false
  • Convert Flickr2K dataset into .t7 files
    cd $makeReposit/NTIRE2017/code/tools
    
    # This command generates multiple t7 files for
    # each image in Flickr2K_HR folder
    th png_to_t7.lua -apath $makeData -dataset Flickr2K -split true

You can also use .png files too. Please see below Training section for the details.

Training

  1. To train our baseline model, please run the following command:

    th main.lua         # This model is not our final model!
    • Here are some optional arguments you can adjust. If you have any problem, please refer following lines. You can check out details in NTIRE2017/code/opts.lua.
      # You can train the model with multiple GPU. (Not multi-scale model.)
      -nGPU       [n]
      
      # Number of threads for data loading.
      -nThreads   [n]   
      
      # Please specify this directory. Default is /var/tmp/dataset
      -datadir    [$makeData]  
      
      # You can make an experiment folder with the name you want.
      -save       [Folder name]
      
      # You can resume your experiment from the last checkpoint.
      # Please do not set -save and -load at the same time.
      -load       [Folder name]     
      
      # png < t7 < t7pack - requires larger memory
      # png > t7 > t7pack - requires faster CPU & Storage
      -datatype   [png | t7 | t7pack]     
      
      # Please increase the splitBatch when you see 'out of memory' during training.
      # S should be the power of 2. (1, 2, 4, ...)
      -splitBatch [S]
      
      # Please reduce the chopSize when you see 'out of memory' during test.
      # The optimal size of S can be vary depend on your maximum GPU memory.
      -chopSize   [S]
  2. To train our EDSR and MDSR, please use the training.sh in NTIRE2017/code directory. You have to uncomment the line you want to execute.

    cd $makeReposit/NTIRE2017/code
    sh training.sh

    Some model may require pre-trained bicubic scale 2 or bicubic multiscale model. Here, we assume that you already downloaded bicubic_x2.t7 and bicubic_multiscale.t7 in the NTIRE2017/demo/model directory. Otherwise, you can create them yourself. It is also possible to start the traning from scratch by removing -preTrained option in training.sh.


Results

result_1

result_2

result_3

result_4

result_5

result_6

result_7

result_8

result_9

result_10

result_11

result_12

result_13

result_14

result_15

result_16

result_17

result_18

result_19

result_20

NTIRE2017 SR Challenge: Unknown Down-sampling Track

unknown_1

unknown_2

Owner
Bee Lim
Bee Lim
Official pytorch implementation of the paper: "SinGAN: Learning a Generative Model from a Single Natural Image"

SinGAN Project | Arxiv | CVF | Supplementary materials | Talk (ICCV`19) Official pytorch implementation of the paper: "SinGAN: Learning a Generative M

Tamar Rott Shaham 3.2k Dec 25, 2022
Code for the paper "PortraitNet: Real-time portrait segmentation network for mobile device" @ CAD&Graphics2019

PortraitNet Code for the paper "PortraitNet: Real-time portrait segmentation network for mobile device". @ CAD&Graphics 2019 Introduction We propose a

265 Dec 01, 2022
Deep learning image registration library for PyTorch

TorchIR: Pytorch Image Registration TorchIR is a image registration library for deep learning image registration (DLIR). I have integrated several ide

Bob de Vos 40 Dec 16, 2022
Systematic generalisation with group invariant predictions

Requirements are Python 3, TensorFlow v1.14, Numpy, Scipy, Scikit-Learn, Matplotlib, Pillow, Scikit-Image, h5py, tqdm. Experiments were run on V100 GPUs (16 and 32GB).

Faruk Ahmed 30 Dec 01, 2022
Source code for Fixed-Point GAN for Cloud Detection

FCD: Fixed-Point GAN for Cloud Detection PyTorch source code of Nyborg & Assent (2020). Abstract The detection of clouds in satellite images is an ess

Joachim Nyborg 8 Dec 22, 2022
A spatial genome aligner for analyzing multiplexed DNA-FISH imaging data.

jie jie is a spatial genome aligner. This package parses true chromatin imaging signal from noise by aligning signals to a reference DNA polymer model

Bojing Jia 9 Sep 29, 2022
Repository for "Improving evidential deep learning via multi-task learning," published in AAAI2022

Improving evidential deep learning via multi task learning It is a repository of AAAI2022 paper, “Improving evidential deep learning via multi-task le

deargen 11 Nov 19, 2022
Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning, CVPR 2021

Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning By Zhenda Xie*, Yutong Lin*, Zheng Zhang, Yue Ca

Zhenda Xie 293 Dec 20, 2022
A set of tools to pre-calibrate and calibrate (multi-focus) plenoptic cameras (e.g., a Raytrix R12) based on the libpleno.

COMPOTE: Calibration Of Multi-focus PlenOpTic camEra. COMPOTE is a set of tools to pre-calibrate and calibrate (multifocus) plenoptic cameras (e.g., a

ComSEE - Computers that SEE 4 May 10, 2022
CoReD: Generalizing Fake Media Detection with Continual Representation using Distillation (ACMMM'21 Oral Paper)

CoReD: Generalizing Fake Media Detection with Continual Representation using Distillation (ACMMM'21 Oral Paper) (Accepted for oral presentation at ACM

Minha Kim 1 Nov 12, 2021
A package for "Procedural Content Generation via Reinforcement Learning" OpenAI Gym interface.

Readme: Illuminating Diverse Neural Cellular Automata for Level Generation This is the codebase used to generate the results presented in the paper av

Sam Earle 27 Jan 05, 2023
Minimal implementation of PAWS (https://arxiv.org/abs/2104.13963) in TensorFlow.

PAWS-TF 🐾 Implementation of Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samples (PAWS)

Sayak Paul 43 Jan 08, 2023
PyTorch Implementation of PortaSpeech: Portable and High-Quality Generative Text-to-Speech

PortaSpeech - PyTorch Implementation PyTorch Implementation of PortaSpeech: Portable and High-Quality Generative Text-to-Speech. Model Size Module Nor

Keon Lee 279 Jan 04, 2023
🦙 LaMa Image Inpainting, Resolution-robust Large Mask Inpainting with Fourier Convolutions, WACV 2022

🦙 LaMa Image Inpainting, Resolution-robust Large Mask Inpainting with Fourier Convolutions, WACV 2022

Advanced Image Manipulation Lab @ Samsung AI Center Moscow 4.7k Dec 31, 2022
Pyramid Pooling Transformer for Scene Understanding

Pyramid Pooling Transformer for Scene Understanding Requirements: torch 1.6+ torchvision 0.7.0 timm==0.3.2 Validated on torch 1.6.0, torchvision 0.7.0

Yu-Huan Wu 119 Dec 29, 2022
StyleGAN - Official TensorFlow Implementation

StyleGAN — Official TensorFlow Implementation Picture: These people are not real – they were produced by our generator that allows control over differ

NVIDIA Research Projects 13.1k Jan 09, 2023
Semi-Supervised Learning for Fine-Grained Classification

Semi-Supervised Learning for Fine-Grained Classification This repo contains the code of: A Realistic Evaluation of Semi-Supervised Learning for Fine-G

25 Nov 08, 2022
A smaller subset of 10 easily classified classes from Imagenet, and a little more French

Imagenette 🎶 Imagenette, gentille imagenette, Imagenette, je te plumerai. 🎶 (Imagenette theme song thanks to Samuel Finlayson) NB: Versions of Image

fast.ai 718 Jan 01, 2023
NU-Wave: A Diffusion Probabilistic Model for Neural Audio Upsampling

NU-Wave: A Diffusion Probabilistic Model for Neural Audio Upsampling For Official repo of NU-Wave: A Diffusion Probabilistic Model for Neural Audio Up

Rishikesh (ऋषिकेश) 38 Oct 11, 2022
A fast, distributed, high performance gradient boosting (GBT, GBDT, GBRT, GBM or MART) framework based on decision tree algorithms, used for ranking, classification and many other machine learning tasks.

Light Gradient Boosting Machine LightGBM is a gradient boosting framework that uses tree based learning algorithms. It is designed to be distributed a

Microsoft 14.5k Jan 08, 2023