The source code of CVPR 2019 paper "Deep Exemplar-based Video Colorization".

Overview

Deep Exemplar-based Video Colorization (Pytorch Implementation)

Paper | Pretrained Model | Youtube video 🔥 | Colab demo

Deep Exemplar-based Video Colorization, CVPR2019

Bo Zhang1,3, Mingming He1,5, Jing Liao2, Pedro V. Sander1, Lu Yuan4, Amine Bermak1, Dong Chen3
1Hong Kong University of Science and Technology,2City University of Hong Kong, 3Microsoft Research Asia, 4Microsoft Cloud&AI, 5USC Institute for Creative Technologies

Prerequisites

  • Python 3.6+
  • Nvidia GPU + CUDA, CuDNN

Installation

First use the following commands to prepare the environment:

conda create -n ColorVid python=3.6
source activate ColorVid
pip install -r requirements.txt

Then, download the pretrained models from this link, unzip the file and place the files into the corresponding folders:

  • video_moredata_l1 under the checkpoints folder
  • vgg19_conv.pth and vgg19_gray.pth under the data folder

Data Preparation

In order to colorize your own video, it requires to extract the video frames, and provide a reference image as an example.

  • Place your video frames into one folder, e.g., ./sample_videos/v32_180
  • Place your reference images into another folder, e.g., ./sample_videos/v32

If you want to automatically retrieve color images, you can try the retrieval algorithm from this link which will retrieve similar images from the ImageNet dataset. Or you can try this link on your own image database.

Test

python test.py --image-size [image-size] \
               --clip_path [path-to-video-frames] \
               --ref_path [path-to-reference] \
               --output_path [path-to-output]

We provide several sample video clips with corresponding references. For example, one can colorize one sample legacy video using:

python test.py --clip_path ./sample_videos/clips/v32 \
               --ref_path ./sample_videos/ref/v32 \
               --output_path ./sample_videos/output

Note that we use 216*384 images for training, which has aspect ratio of 1:2. During inference, we scale the input to this size and then rescale the output back to the original size.

Train

We also provide training code for reference. The training can be started by running:

python --data_root [root of video samples] \
       --data_root_imagenet [root of image samples] \
       --gpu_ids [gpu ids] \

We do not provide the full video dataset due to the copyright issue. For image samples, we retrieve semantically similar images from ImageNet using this repository. Still, one can refer to our code to understand the detailed procedure of augmenting the image dataset to mimic the video frames.

Comparison with State-of-the-Arts

More results

Please check our Youtube demo for results of video colorization.

Citation

If you use this code for your research, please cite our paper.

@inproceedings{zhang2019deep,
title={Deep exemplar-based video colorization},
author={Zhang, Bo and He, Mingming and Liao, Jing and Sander, Pedro V and Yuan, Lu and Bermak, Amine and Chen, Dong},
booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
pages={8052--8061},
year={2019}
}

Old Photo Restoration 🔥

If you are also interested in restoring the artifacts in the legacy photo, please check our recent work, bringing old photo back to life.

@inproceedings{wan2020bringing,
title={Bringing Old Photos Back to Life},
author={Wan, Ziyu and Zhang, Bo and Chen, Dongdong and Zhang, Pan and Chen, Dong and Liao, Jing and Wen, Fang},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={2747--2757},
year={2020}
}

License

This project is licensed under the MIT license.

Comments
  • I met

    I met "CUDA error: an illegal memory access was encountered" problem

    When i tested your pre-trained model, i met the problem called "CUDA error: an illegal memory access was encountered", can you provide the version of your CUDA, cudnn and pytorch

    opened by Zonobia-A 3
  • Could i apply this method to image colorization

    Could i apply this method to image colorization

    Hi, could i apply this method to image colorization and remove the temporal consistency loss? BTW, how to get the pairs.txt/pairs_mid.txt/pairs_bad.txt used in videoloader_imagenet.py?

    opened by buptlj 2
  • video size very low

    video size very low

    video colorization very good and very impressive . but render image low size 768x432 . and video size also same. how to in increase the image size and video size. thank you..

    opened by srinivas68 2
  • training command is wrong.

    training command is wrong.

    The original training command is python --data_root [root of video samples] \ --data_root_imagenet [root of image samples] \ --gpu_ids [gpu ids] \ Maybe it should be python train.py --data_root [root of video samples] \ --data_root_imagenet [root of image samples] \ --gpu_ids [gpu ids] \ ?

    opened by Horizon2333 1
  • There seems a bug ofr feature centering with x_features - y_features.mean

    There seems a bug ofr feature centering with x_features - y_features.mean

    There seems a bug ofr feature centering with x_features - y_features.mean which I think should be x_features - x_features.mean https://github.com/zhangmozhe/Deep-Exemplar-based-Video-Colorization/blob/37639748f12dfecbb0a3fe265b533887b5fe46ce/models/ContextualLoss.py#L100

    opened by JerryLeolfl 1
  • the test code

    the test code

    Thanks for your great work! I have a question when i run test.py. Why don't you extract the feature of inference image out of the for loop. I haven't found any difference.

    opened by buptlj 1
  • CUDA OOM

    CUDA OOM

    Hello, I am running a 4gb nvidia GPU. Is that enough for inference? I try to run on ubuntu 18.04 as well as windows but always get a Out of memory error eventually. Sometimes happen after 2nd image and sometimes after 5th. This is 1080p video.

    opened by quocthaitang 1
  • illustrative training data

    illustrative training data

    Could you please release a tiny illustrative training dataset, such that the preparation of a custom training data can be easily followed. Currently, it is not easy to prepare a custom training data by reading the train.py. or could you please give a further explanation of the following fields? ( image1_name, image2_name, reference_video_name, reference_video_name1, reference_name1, reference_name2, reference_name3, reference_name4, reference_name5, reference_gt1, reference_gt2, reference_gt3, ) Thank you very much.

    opened by davyfeng 1
  • Runtime error

    Runtime error

    Getting a runtime error when running the test cells at the 'first we visualize the input video'. I'm not good with code, but this is the first time I've experienced this issue with this wonderful program. No cells execute after this error. I've attached a screenshot. IMG_7258

    opened by StevieMaccc 0
  • Test result problem

    Test result problem

    At present, after training, it is found that the generated test image is effective, but the color saturation is very low. Is it because of the colored model or other reasons? I'm looking forward to your reply!!!

    opened by songyn95 0
  • Training has little effect

    Training has little effect

    Hello, I read in the paper that "we train the network for 10 epichs with a batch size of 40 pairs of video frames. " Is it effective after only 10 iterations? Is your data 768 videos, 25 frames per video? I only train one video at present, epoch=40, but I find that it has little effect. What may be the reason?

    opened by songyn95 0
  • Error 404 - Important files missing

    Error 404 - Important files missing

    I was working with the Colab program and there appears to be important models / files missing. As a result the program has ceased to function. I've brough to the designers attention so hopefully will be resolved.

    opened by StevieMaccc 1
  • CUDA device error

    CUDA device error "module 'torch._C' has no attribute '_cuda_setDevice'" when running test.py

    Hi !

    Trying out test.py results in the following error:

    Traceback (most recent call last): File "test.py", line 26, in <module> torch.cuda.set_device(0) File "C:\Users\natha\anaconda3\envs\ColorVid\lib\site-packages\torch\cuda\__init__.py", line 311, in set_device torch._C._cuda_setDevice(device) AttributeError: module 'torch._C' has no attribute '_cuda_setDevice'

    I tried installing pytorch manually using their tool https://pytorch.org/get-started/locally/ (with CUDA 11.6) but that doesn't resolve the issue.

    Can someone help me understand what is going on ? Thanks !!

    opened by FoxTrotte 4
  • Questions about the test phase

    Questions about the test phase

    Thanks for your outstanding work! I have some questions when I read it.

    1. What are the settings when you test this video model on image colorization which used for comparing with other image colorization methods?
    2. Could you please give me a url about your video testset (116 video clips collected from Videvo)? Thanks again for your attention.
    opened by JerryLeolfl 0
  • It seems not correct of the code in TestTransforms.py line 341

    It seems not correct of the code in TestTransforms.py line 341

    https://github.com/zhangmozhe/Deep-Exemplar-based-Video-Colorization/blob/37639748f12dfecbb0a3fe265b533887b5fe46ce/lib/TestTransforms.py#L341 it seems a repeated define of call

    opened by JerryLeolfl 0
  • Wrong output resolution

    Wrong output resolution

    Processing 4x3 video 912x720 outputs cropped and downscaled 16x9 768x432. Playing around "python test.py --image-size [image-size] " doesn't help My be I don't properly specify an arguments? So, what the the proper use of --image-size [image-size] in order to get 912x720? Greatly appreciate for suggesting.

    opened by semel1 5
Segmentation Training Pipeline

Segmentation Training Pipeline This package is a part of Musket ML framework. Reasons to use Segmentation Pipeline Segmentation Pipeline was developed

Musket ML 52 Dec 12, 2022
Video-Music Transformer

VMT Video-Music Transformer (VMT) is an attention-based multi-modal model, which generates piano music for a given video. Paper https://arxiv.org/abs/

Chin-Tung Lin 5 Jul 13, 2022
免费获取http代理并生成proxifier配置文件

freeproxy 免费获取http代理并生成proxifier配置文件 公众号:台下言书 工具说明:https://mp.weixin.qq.com/s?__biz=MzIyNDkwNjQ5Ng==&mid=2247484425&idx=1&sn=56ccbe130822aa35038095317

说书人 32 Mar 25, 2022
Solutions and questions for AoC2021. Merry christmas!

Advent of Code 2021 Merry christmas! 🎄 🎅 To get solutions and approximate execution times for implementations, please execute the run.py script in t

Wilhelm Ågren 5 Dec 29, 2022
Local Attention - Flax module for Jax

Local Attention - Flax Autoregressive Local Attention - Flax module for Jax Install $ pip install local-attention-flax Usage from jax import random fr

Phil Wang 16 Jun 16, 2022
RTS3D: Real-time Stereo 3D Detection from 4D Feature-Consistency Embedding Space for Autonomous Driving

RTS3D: Real-time Stereo 3D Detection from 4D Feature-Consistency Embedding Space for Autonomous Driving (AAAI2021). RTS3D is efficiency and accuracy s

71 Nov 29, 2022
End-to-end face detection, cropping, norm estimation, and landmark detection in a single onnx model

onnx-facial-lmk-detector End-to-end face detection, cropping, norm estimation, and landmark detection in a single onnx model, model.onnx. Demo You can

atksh 42 Dec 30, 2022
yolov5 deepsort 行人 车辆 跟踪 检测 计数

yolov5 deepsort 行人 车辆 跟踪 检测 计数 实现了 出/入 分别计数。 默认是 南/北 方向检测,若要检测不同位置和方向,可在 main.py 文件第13行和21行,修改2个polygon的点。 默认检测类别:行人、自行车、小汽车、摩托车、公交车、卡车。 检测类别可在 detect

554 Dec 30, 2022
This is the code for the paper "Jinkai Zheng, Xinchen Liu, Wu Liu, Lingxiao He, Chenggang Yan, Tao Mei: Gait Recognition in the Wild with Dense 3D Representations and A Benchmark. (CVPR 2022)"

Gait3D-Benchmark This is the code for the paper "Jinkai Zheng, Xinchen Liu, Wu Liu, Lingxiao He, Chenggang Yan, Tao Mei: Gait Recognition in the Wild

82 Jan 04, 2023
Code, environments, and scripts for the paper: "How Private Is Your RL Policy? An Inverse RL Based Analysis Framework"

Privacy-Aware Inverse RL (PRIL) Analysis Framework Code, environments, and scripts for the paper: "How Private Is Your RL Policy? An Inverse RL Based

1 Dec 06, 2021
Implementation for "Manga Filling Style Conversion with Screentone Variational Autoencoder" (SIGGRAPH ASIA 2020 issue)

Manga Filling with ScreenVAE SIGGRAPH ASIA 2020 | Project Website | BibTex This repository is for ScreenVAE introduced in the following paper "Manga F

30 Dec 24, 2022
High-performance moving least squares material point method (MLS-MPM) solver.

High-Performance MLS-MPM Solver with Cutting and Coupling (CPIC) (MIT License) A Moving Least Squares Material Point Method with Displacement Disconti

Yuanming Hu 2.2k Dec 31, 2022
[3DV 2020] PeeledHuman: Robust Shape Representation for Textured 3D Human Body Reconstruction

PeeledHuman: Robust Shape Representation for Textured 3D Human Body Reconstruction International Conference on 3D Vision, 2020 Sai Sagar Jinka1, Rohan

Rohan Chacko 39 Oct 12, 2022
Official implementation of "Refiner: Refining Self-attention for Vision Transformers".

RefinerViT This repo is the official implementation of "Refiner: Refining Self-attention for Vision Transformers". The repo is build on top of timm an

101 Dec 29, 2022
Compressed Video Action Recognition

Compressed Video Action Recognition Chao-Yuan Wu, Manzil Zaheer, Hexiang Hu, R. Manmatha, Alexander J. Smola, Philipp Krähenbühl. In CVPR, 2018. [Proj

Chao-Yuan Wu 479 Dec 26, 2022
Official repository for the paper "Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks"

Easy-To-Hard The official repository for the paper "Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks". Gett

Avi Schwarzschild 52 Sep 08, 2022
Template repository to build PyTorch projects from source on any version of PyTorch/CUDA/cuDNN.

The Ultimate PyTorch Source-Build Template Translations: 한국어 TL;DR PyTorch built from source can be x4 faster than a naïve PyTorch install. This repos

Joonhyung Lee/이준형 651 Dec 12, 2022
Pytorch implementation for DFN: Distributed Feedback Network for Single-Image Deraining.

DFN:Distributed Feedback Network for Single-Image Deraining Abstract Recently, deep convolutional neural networks have achieved great success for sing

6 Nov 05, 2022
From this paper "SESNet: A Semantically Enhanced Siamese Network for Remote Sensing Change Detection"

SESNet for remote sensing image change detection It is the implementation of the paper: "SESNet: A Semantically Enhanced Siamese Network for Remote Se

1 May 24, 2022