NeRV: Neural Representations for Videos (NeurIPS 2021)
Project Page | Paper | UVG Data
Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser-Nam Lim, Abhinav Shrivastava
This is the official implementation of the paper "NeRV: Neural Representations for Videos ".
Get started
We run with Python 3.8, you can set up a conda environment with all dependencies like so:
pip install -r requirements.txt
High-Level structure
The code is organized as follows:
- train_nerv.py includes a generic traiing routine.
- model_nerv.py contains the dataloader and neural network architecure
- data/ directory video/imae dataset, we provide big buck bunny here
- checkpoint/ directory contains some pre-trained model on big buck bunny dataset
- log files (tensorboard, txt, state_dict etc.) will be saved in output directory (specified by
--outf)
Reproducing experiments
Training experiments
The NeRV-S experiment on 'big buck bunny' can be reproduced with
python train_nerv.py -e 300 --cycles 1 --lower-width 96 --num-blocks 1 --dataset bunny --frame_gap 1 \
--outf bunny_ab --embed 1.25_40 --stem_dim_num 512_1 --reduction 2 --fc_hw_dim 9_16_26 --expansion 1 \
--single_res --loss Fusion6 --warmup 0.2 --lr_type cosine --strides 5 2 2 2 2 --conv_type conv \
-b 1 --lr 0.0005 --norm none --act swish
Evaluation experiments
To evaluate pre-trained model, just add --eval_Only and specify model path with --weight, you can specify model quantization with --quant_bit [bit_lenght], yuo can test decoding speed with --eval_fps, below we preovide sample commends for NeRV-S on bunny dataset
python train_nerv.py -e 300 --cycles 1 --lower-width 96 --num-blocks 1 --dataset bunny --frame_gap 1 \
--outf bunny_ab --embed 1.25_40 --stem_dim_num 512_1 --reduction 2 --fc_hw_dim 9_16_26 --expansion 1 \
--single_res --loss Fusion6 --warmup 0.2 --lr_type cosine --strides 5 2 2 2 2 --conv_type conv \
-b 1 --lr 0.0005 --norm none --act swish \
--weight checkpoints/nerv_S.pth --eval_only
Dump predictions with pre-trained model
To evaluate pre-trained model, just add --eval_Only and specify model path with --weight
python train_nerv.py -e 300 --cycles 1 --lower-width 96 --num-blocks 1 --dataset bunny --frame_gap 1 \
--outf bunny_ab --embed 1.25_40 --stem_dim_num 512_1 --reduction 2 --fc_hw_dim 9_16_26 --expansion 1 \
--single_res --loss Fusion6 --warmup 0.2 --lr_type cosine --strides 5 2 2 2 2 --conv_type conv \
-b 1 --lr 0.0005 --norm none --act swish \
--weight checkpoints/nerv_S.pth --eval_only --dump_images
Citation
If you find our work useful in your research, please cite:
@inproceedings{hao2021nerv,
author = {Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser-Nam Lim, Abhinav Shrivastava },
title = {NeRV: Neural Representations for Videos s},
booktitle = {NeurIPS},
year={2021}
}
Contact
If you have any questions, please feel free to email the authors.
53 Jan 08, 2023
11.3k Dec 31, 2022
47 Dec 22, 2022
8 Jun 07, 2022
94 Dec 23, 2022
7 Oct 31, 2022
0 Aug 01, 2022
352 Dec 25, 2022
0 Jan 27, 2022
18 Dec 19, 2022
7.3k Jan 07, 2023
36 Oct 18, 2022
19 Nov 11, 2022
12 Oct 21, 2022
378 Dec 30, 2022
2.5k Jan 02, 2023
24 Dec 02, 2022
1 Feb 09, 2022
7.7k Jan 05, 2023
369 Dec 24, 2022