Implementation of Research Paper "Learning to Enhance Low-Light Image via Zero-Reference Deep Curve Estimation"

Overview

Zero-DCE and Zero-DCE++(Lite architechture for Mobile and edge Devices)

TensorFlow Keras Python

PWC PWC PWC PWC PWC

GitHub license GitHub stars GitHub forks GitHub watchers

Papers Abstract

The paper presents a novel method, Zero-Reference Deep Curve Estimation (Zero-DCE), which formulates 
light enhancement as a task of image-specific curve estimation with a deep network. 
Our method trains a lightweight deep network, DCE-Net, to estimate pixel-wise and 
high-order curves for dynamic range adjustment of a given image. The curve estimation 
is specially designed, considering pixel value range, monotonicity, and differentiability. 
Zero-DCE is appealing in its relaxed assumption on reference images, i.e., it does not 
require any paired or unpaired data during training. This is achieved through a set of 
carefully formulated non-reference loss functions, which implicitly measure the 
enhancement quality and drive the learning of the network. Our method is efficient 
as image enhancement can be achieved by an intuitive and simple nonlinear curve mapping. 
Despite its simplicity, we show that it generalizes well to diverse lighting conditions. 
Extensive experiments on various benchmarks demonstrate the advantages of our method over 
state-of-the-art methods qualitatively and quantitatively. Furthermore, the potential benefits
of our Zero-DCE to face detection in the dark are discussed. We further present an 
accelerated and light version of Zero-DCE, called (Zero-DCE++), that takes advantage 
of a tiny network with just 10K parameters. Zero-DCE++ has a fast inference speed 
(1000/11 FPS on single GPU/CPU for an image with a size of 1200*900*3) while keeping 
the enhancement performance of Zero-DCE.

๐Ÿ“œ Paper link: Zero-Reference Deep Curve Estimation (Zero-DCE)

๐Ÿ“œ Paper link: Learning to Enhance Low-Light Image via Zero-Reference Deep Curve Estimation (Zero-DCE++)

Check out the original Pytorch Implementation of Zero-DCE here and the original Pytorch implementation of Zero-DCE++ here

Proposed Zero-DCE Framework

Proposed Zero-DCE Framework

The paper proposed a Zero Reference(without a label/reference image) Deep Curve Estimation network which estimates the best-fitting Light-Enhancement curve (LE-Curve) for a given image. Further the framework then maps all the pixels of the image's RGB channels by applying the best-fit curve iteratively and get the final enhanced output image.

DCE-net and DCE-net++

DCE-net architecture

The paper proposes a simple CNN bases Deep neural network called DCE-net, which learns to map the input low-light image to its best-fit curve parameters maps. The network consist of 7 convolution layers with symmetrical skip concatenation. First 6 convolution layers consist of 32 filters each with kernel size of 3x3 with stride of 1 followed by RelU activation. The last convolution layer has interation x 3 number of filters (if we set iteration to 8 it will produce 24 curve parameters maps for 8 iteration, where each iteration generates three curve parameter maps for the three RGB channels) followed by tanh activation. The proposed DCE-net architechture does not contains any max-pooling, downsampling or batch-normalization layers as it can break the relations between neighboring pixels.

DCE-net++ is the lite version of DCE-net. DCE-net is already a very light model with just 79k parameters. The main changes in DCE-net++ are:

  1. Instead of traditional convolutional layers, we use Depthwise separable convolutional layers which significantly reduces the total number of parameters, uses less memory and computational power. The DCE-net++ architecture has a total of 10k parameters with same architecture design as DCE-net.
  2. The last convolution layers has only 3 filters instead of interation x 3 number of filters which can be used to iteratively enhance the images.

Zero-Reference Loss Functions

The paper proposes set of zero-reference loss functions that differntiable which allows to assess the quality of enhanced image.

  1. Spatial Consistency Loss The spatial consistency loss $L_{spa}$ encourages spatial coherence of the enhanced image through preserving the difference of neighboring regions between the input image and its enhanced version

    Spatial Consistency loss

  2. Exposure controll loss To restrain the exposure of the enhanced image, the exposure control loss $L_{exp}$ is designed to control the exposure of the enhanced image. The exposure control loss measures the distance between the average intensity value of a local region to the well-exposedness level $E$.

Exposure control loss

  1. Color Constancy loss By Following the Gray-world hypothesis that color in each sensor channel(RGB) averages to gray over the entire image, the paper proposes a color constancy loss $L_{col}$ to correct the potential diviation of color in the enhanced image.

    Color Constancy loss

  2. Illumination Smoothness Loss To preserve the monotonicity relations between neighboring pixels, we add an illumination smoothness loss to each curve parameter map A.

    Illumination Smoothness loss

Training and Testing Model

Zero-DCE and Zero-DCE++ model was created using Tensorflow 2.7.0 and Keras and trained on google colab's Tesla K80 GPU (12GB VRAM)

Dataset pipeline and Dataset used

I used Tensorflow's tf.data api to create a dataset input pipeline. Input data pipeline

dataset structure:

lol_datasetv2
โ”œโ”€โ”€ 100.png
โ”œโ”€โ”€ 101.png
โ”œโ”€โ”€ 102.png
โ”œโ”€โ”€ 103.png
โ”œโ”€โ”€ 109.png
โ”œโ”€โ”€ 10.png
โ”œโ”€โ”€ 95.png
โ”œโ”€โ”€ 96.png
โ”œโ”€โ”€ 97.png
โ”œโ”€โ”€ 98.png
โ”œโ”€โ”€ 99.png
โ””โ”€โ”€ 9.png

0 directories, 500 files

Dataset link: LoL-dataset

Usage

  • Clone this github repo
  • Run $pip install -r requirements.txt to install required python packgages.

For training the model, run following

$ python train_model.py --help
usage: train_model.py [-h] --dataset_dir DATASET_DIR [--checkpoint_dir CHECKPOINT_DIR] [--model_type MODEL_TYPE] [--IMG_H IMG_H]
                      [--IMG_W IMG_W] [--IMG_C IMG_C] [--batch_size BATCH_SIZE] [--epoch EPOCH] [--learning_rate LEARNING_RATE]
                      [--dataset_split DATASET_SPLIT] [--logdir LOGDIR] [--iteration ITERATION]

Model training scipt for Zero-DCE models

optional arguments:
  -h, --help            show this help message and exit
  --dataset_dir DATASET_DIR
                        Dataset directory
  --checkpoint_dir CHECKPOINT_DIR
                        Checkpoint directory
  --model_type MODEL_TYPE
                        Type of Model.should be any of: ['zero_dce', 'zero_dce_lite']
  --IMG_H IMG_H         Image height
  --IMG_W IMG_W         Image width
  --IMG_C IMG_C         Image channels
  --batch_size BATCH_SIZE
                        Batch size
  --epoch EPOCH         Epochs
  --learning_rate LEARNING_RATE
                        Learning rate
  --dataset_split DATASET_SPLIT
                        Dataset split
  --logdir LOGDIR       Log directory
  --iteration ITERATION
                        Post enhancing iteration

Example

!python train_model.py --dataset_dir lol_datasetv2/ \
                      --model_type zero_dce_lite \
                      --checkpoint_dir Trained_model/ \ 
                      --IMG_H 512 \
                      --IMG_W 512 \
                      --epoch 60 \
                      --batch_size 4 \ 
                      --iteration 6 \

Testing the model on the test dataset

$ python test_model.py --help                                                                                                                    
usage: test_model.py [-h] --model_path MODEL_PATH [--dataset_path DATASET_PATH] [--img_h IMG_H] [--img_w IMG_W] [--save_plot SAVE_PLOT]
                     [--load_random_data LOAD_RANDOM_DATA]

Test model on test dataset

optional arguments:
  -h, --help            show this help message and exit
  --model_path MODEL_PATH
                        path to the saved model folder
  --dataset_path DATASET_PATH
                        path to the dataset
  --img_h IMG_H         image height
  --img_w IMG_W         Image width
  --save_plot SAVE_PLOT
                        save plot of original vs enhanced image. 0: no, 1: yes
  --load_random_data LOAD_RANDOM_DATA
                        load random data. 0: no, 1: yes

Example

!python test_model.py --model_path Trained_model/zero_dce_lite_iter8/zero_dce_lite_200x300_iter8_60/ \
                      --datset_path lol_datasetv2/ \
                      --img_h 200 \
                      --img_w 300 \
                      --save_plot 1 \
                      --load_random_data 0

Inferencing on single image for enhancement

$ python single_image_enhance.py --help                                                                                      
usage: single_image_enhance.py [-h] --model_path MODEL_PATH --image_path IMAGE_PATH [--img_h IMG_H] [--img_w IMG_W] [--plot PLOT] [--save_result SAVE_RESULT] [--iteration ITERATION]

Single Image Enhancement

optional arguments:
  -h, --help            show this help message and exit
  --model_path MODEL_PATH
                        path to tf model
  --image_path IMAGE_PATH
                        path to image file
  --img_h IMG_H         image height
  --img_w IMG_W         image width
  --plot PLOT           plot enhanced image
  --save_result SAVE_RESULT
                        save enhanced image
  --iteration ITERATION
                        number of Post Ehnancing iterations

Example

$ python single_image_enhance.py --model_path Trained_model/zero_dce_iter6/zero_dce_200x300_iter6_30 \
                                --img_h 200 \
                                --img_w 300 \
                                --image_path sample_images/ low_light_outdoor.jpg \
                                --plot 0 \
                                --save_result 1 \
                                --iteration 6 \

Visual Results

Testset Results

1.Model: Zero-DCE, Epoch:30 , Input size:200x300, Iteration:4, Average Time: CPU-170.0 ms

test_image_plot_zero_dce_iter4_30

2.Model: Zero-DCE, Epoch:30, Input size: 200x300, Iteration:6, Average Time: CPU-170.0 ms

test_image_plot_zero_dce_iter6_30.png

3.Model: Zero-DCE, Epoch:30, Inout size: 200x300, Iteration:8, Average Time: CPU-170.0 ms

test_image_plot_zero_dce_iter8_30

4.Model: Zero-DCE Lite, Epoch:60, Input size: 512x512, Iteration:6, Average Time: CPU-450 ms

test_image_plot_zero_dce_lite_iter6

5.Model: Zero-DCE Lite, Epoch:60, Input size: 200x300, Iteration:8, Average Time: CPU-90 ms

test_image_plot_zero_dce_lite_iter8

Enhance Image with its Alpha Maps.(Curve Parameter Maps)

enhanced_result_with_alpha_maps_zero_dce_100

enhanced_result_with_alpha_maps_zero_dce_512x512_e_60

Test Results on out of dataset images

img img
low light image Enhanced Image(Zero-DCE, epoch:60, interation:4)
img img
low light image Enhanced Image(Zero-DCE, epoch:60, interation:6)
img img
low light image Enhanced Image(Zero-DCE, epoch:30, interation:8)
img img
low light image Enhanced Image(Zero-DCE, epoch:30, interation:6)
img img
low light image Enhanced Image(Zero-DCE lite, epoch:60, interation:8)
img img
low light image Enhanced Image(Zero-DCE, epoch:30, interation:8)
img img
low light image Enhanced Image(Zero-DCE lite, epoch:60, interation:8)
img img
low light image Enhanced Image(Zero-DCE lite, epoch:60, interation:6)

Best SavedModel for Zero-DCE and Zero-DCE Lite

Releasing soon

Demo Apllication

Mobile Demo application of our trained model is comming soon

References

Citation

Paper: Zero-DCE

@Article{Zero-DCE,
          author = {Guo, Chunle and Li, Chongyi and Guo, Jichang and Loy, Chen Change and Hou, 
                    Junhui and Kwong, Sam and Cong Runmin},
          title = {Zero-reference deep curve estimation for low-light image enhancement},
          journal = {CVPR},
          pape={1780-1789},
          year = {2020}
    }

Paper: Zero-DCE++

@Article{Zero-DCE++,
          author ={Li, Chongyi and Guo, Chunle and Loy, Chen Change},
          title = {Learning to Enhance Low-Light Image via Zero-Reference Deep Curve Estimation},
          journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
          pape={},
          year = {2021},
          doi={10.1109/TPAMI.2021.3063604}
          }

Dataset

@inproceedings{Chen2018Retinex,

  title={Deep Retinex Decomposition for Low-Light Enhancement},

  author={Chen Wei, Wenjing Wang, Wenhan Yang, Jiaying Liu},

  booktitle={British Machine Vision Conference},

  year={2018},

} 
You might also like...
The lightweight PyTorch wrapper for high-performance AI research. Scale your models, not the boilerplate.
The lightweight PyTorch wrapper for high-performance AI research. Scale your models, not the boilerplate.

The lightweight PyTorch wrapper for high-performance AI research. Scale your models, not the boilerplate. Website โ€ข Key Features โ€ข How To Use โ€ข Docs โ€ข

A research toolkit for particle swarm optimization in Python
A research toolkit for particle swarm optimization in Python

PySwarms is an extensible research toolkit for particle swarm optimization (PSO) in Python. It is intended for swarm intelligence researchers, practit

Plato: A New Framework for Federated Learning Research

a new software framework to facilitate scalable federated learning research.

Research shows Google collects 20x more data from Android than Apple collects from iOS. Block this non-consensual telemetry using pihole blocklists.

pihole-antitelemetry Research shows Google collects 20x more data from Android than Apple collects from iOS. Block both using these pihole lists. Proj

A Research-oriented Federated Learning Library and Benchmark Platform for Graph Neural Networks. Accepted to ICLR'2021 - DPML and MLSys'21 - GNNSys workshops.

FedGraphNN: A Federated Learning System and Benchmark for Graph Neural Networks A Research-oriented Federated Learning Library and Benchmark Platform

This repository contains the implementations related to the experiments of a set of publicly available datasets that are used in the time series forecasting research space.

TSForecasting This repository contains the implementations related to the experiments of a set of publicly available datasets that are used in the tim

This is the research repository for Vid2Doppler: Synthesizing Doppler Radar Data from Videos for Training Privacy-Preserving Activity Recognition.
This is the research repository for Vid2Doppler: Synthesizing Doppler Radar Data from Videos for Training Privacy-Preserving Activity Recognition.

Vid2Doppler: Synthesizing Doppler Radar Data from Videos for Training Privacy-Preserving Activity Recognition This is the research repository for Vid2

Automatic voice-synthetised summaries of latest research papers on arXiv

PaperWhisperer PaperWhisperer is a Python application that keeps you up-to-date with research papers. How? It retrieves the latest articles from arXiv

A Dataset of Python Challenges for AI Research

Python Programming Puzzles (P3) This repo contains a dataset of python programming puzzles which can be used to teach and evaluate an AI's programming

Releases(v0.1.0)
Owner
Tauhid Khan
Python, ML, DL, Computer Vision.
Tauhid Khan
This example implements the end-to-end MLOps process using Vertex AI platform and Smart Analytics technology capabilities

MLOps with Vertex AI This example implements the end-to-end MLOps process using Vertex AI platform and Smart Analytics technology capabilities. The ex

Google Cloud Platform 238 Dec 21, 2022
Implementation for the paper 'YOLO-ReT: Towards High Accuracy Real-time Object Detection on Edge GPUs'

YOLO-ReT This is the original implementation of the paper: YOLO-ReT: Towards High Accuracy Real-time Object Detection on Edge GPUs. Prakhar Ganesh, Ya

69 Oct 19, 2022
[NeurIPS 2021]: Are Transformers More Robust Than CNNs? (Pytorch implementation & checkpoints)

Are Transformers More Robust Than CNNs? Pytorch implementation for NeurIPS 2021 Paper: Are Transformers More Robust Than CNNs? Our implementation is b

Yutong Bai 145 Dec 01, 2022
Python wrapper of LSODA (solving ODEs) which can be called from within numba functions.

numbalsoda numbalsoda is a python wrapper to the LSODA method in ODEPACK, which is for solving ordinary differential equation initial value problems.

Nick Wogan 52 Jan 09, 2023
Yolov5-lite - Minimal PyTorch implementation of YOLOv5

Yolov5-Lite: Minimal YOLOv5 + Deep Sort Overview This repo is a shortened versio

Kadir Nar 57 Nov 28, 2022
PyTorch3D is FAIR's library of reusable components for deep learning with 3D data

Introduction PyTorch3D provides efficient, reusable components for 3D Computer Vision research with PyTorch. Key features include: Data structure for

Facebook Research 6.8k Jan 01, 2023
Turning pixels into virtual points for multimodal 3D object detection.

Multimodal Virtual Point 3D Detection Turning pixels into virtual points for multimodal 3D object detection. Multimodal Virtual Point 3D Detection, Ti

Tianwei Yin 204 Jan 08, 2023
A PyTorch implementation of the architecture of Mask RCNN

EDIT (AS OF 4th NOVEMBER 2019): This implementation has multiple errors and as of the date 4th, November 2019 is insufficient to be utilized as a reso

Sai Himal Allu 975 Dec 30, 2022
Facilitates implementing deep neural-network backbones, data augmentations

Introduction Nowadays, the training of Deep Learning models is fragmented and unified. When AI engineers face up with one specific task, the common wa

40 Dec 29, 2022
Roach: End-to-End Urban Driving by Imitating a Reinforcement Learning Coach

CARLA-Roach This is the official code release of the paper End-to-End Urban Driving by Imitating a Reinforcement Learning Coach by Zhejun Zhang, Alexa

Zhejun Zhang 118 Dec 28, 2022
Neural network-based build time estimation for additive manufacturing

Neural network-based build time estimation for additive manufacturing Oh, Y., Sharp, M., Sprock, T., & Kwon, S. (2021). Neural network-based build tim

Yosep 1 Nov 15, 2021
Attentive Implicit Representation Networks (AIR-Nets)

Attentive Implicit Representation Networks (AIR-Nets) Preprint | Supplementary | Accepted at the International Conference on 3D Vision (3DV) teaser.mo

29 Dec 07, 2022
Code for the AAAI 2022 paper "Zero-Shot Cross-Lingual Machine Reading Comprehension via Inter-Sentence Dependency Graph".

multilingual-mrc-isdg Code for the AAAI 2022 paper "Zero-Shot Cross-Lingual Machine Reading Comprehension via Inter-Sentence Dependency Graph". This r

Liyan 5 Dec 07, 2022
Gesture Volume Control v.2

Gesture volume control v.2 In this project I am going to learn how to use Gesture Control to change the volume of a computer. I first look into hand t

Pavel Dat 23 Dec 26, 2022
Pytorch implementation of the paper "Topic Modeling Revisited: A Document Graph-based Neural Network Perspective"

Graph Neural Topic Model (GNTM) This is the pytorch implementation of the paper "Topic Modeling Revisited: A Document Graph-based Neural Network Persp

Dazhong Shen 8 Sep 14, 2022
The devkit of the nuPlan dataset.

The devkit of the nuPlan dataset.

Motional 264 Jan 03, 2023
Learn about Spice.ai with in-depth samples

Samples Learn about Spice.ai with in-depth samples ServerOps - Learn when to run server maintainance during periods of low load Gardener - Intelligent

Spice.ai 16 Mar 23, 2022
The project was to detect traffic signs, based on the Megengine framework.

trafficsign ่ต›้ข˜ ๆ—ท่ง†AIๆ™บๆ…งไบค้€šๅผ€ๆบ่ต›้“๏ผŒๅˆ่ต›1/177๏ผŒๅค่ต›1/12ใ€‚ ๆœฌ่ต›้ข˜ไธบๅคๆ‚ๅœบๆ™ฏ็š„ไบค้€šๆ ‡ๅฟ—ๆฃ€ๆต‹๏ผŒๅฏนไบ”็งไบค้€šๆ ‡ๅฟ—่ฟ›่กŒ่ฏ†ๅˆซใ€‚ ๆก†ๆžถ megengine ็ฎ—ๆณ•ๆ–นๆกˆ ็ฝ‘็ปœๆก†ๆžถ atss + resnext101_32x8d ่ฎญ็ปƒ้˜ถๆฎต ๅ›พ็‰‡ๅฐบๅฏธ ๆœ€็ปˆๆไบค็‰ˆๆœฌ่พ“ๅ…ฅๅ›พ็‰‡ๅฐบๅฏธไธบ(1500,2

20 Dec 02, 2022
CRLT: A Unified Contrastive Learning Toolkit for Unsupervised Text Representation Learning

CRLT: A Unified Contrastive Learning Toolkit for Unsupervised Text Representation Learning This repository contains the code and relevant instructions

XiaoMing 5 Aug 19, 2022
SGPT: Multi-billion parameter models for semantic search

SGPT: Multi-billion parameter models for semantic search This repository contains code, results and pre-trained models for the paper SGPT: Multi-billi

Niklas Muennighoff 182 Dec 29, 2022