NeRV: Neural Representations for Videos (NeurIPS 2021)
Project Page | Paper | UVG Data
Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser-Nam Lim, Abhinav Shrivastava
This is the official implementation of the paper "NeRV: Neural Representations for Videos ".
Get started
We run with Python 3.8, you can set up a conda environment with all dependencies like so:
pip install -r requirements.txt
High-Level structure
The code is organized as follows:
- train_nerv.py includes a generic traiing routine.
- model_nerv.py contains the dataloader and neural network architecure
- data/ directory video/imae dataset, we provide big buck bunny here
- checkpoint/ directory contains some pre-trained model on big buck bunny dataset
- log files (tensorboard, txt, state_dict etc.) will be saved in output directory (specified by
--outf)
Reproducing experiments
Training experiments
The NeRV-S experiment on 'big buck bunny' can be reproduced with
python train_nerv.py -e 300 --cycles 1 --lower-width 96 --num-blocks 1 --dataset bunny --frame_gap 1 \
--outf bunny_ab --embed 1.25_40 --stem_dim_num 512_1 --reduction 2 --fc_hw_dim 9_16_26 --expansion 1 \
--single_res --loss Fusion6 --warmup 0.2 --lr_type cosine --strides 5 2 2 2 2 --conv_type conv \
-b 1 --lr 0.0005 --norm none --act swish
Evaluation experiments
To evaluate pre-trained model, just add --eval_Only and specify model path with --weight, you can specify model quantization with --quant_bit [bit_lenght], yuo can test decoding speed with --eval_fps, below we preovide sample commends for NeRV-S on bunny dataset
python train_nerv.py -e 300 --cycles 1 --lower-width 96 --num-blocks 1 --dataset bunny --frame_gap 1 \
--outf bunny_ab --embed 1.25_40 --stem_dim_num 512_1 --reduction 2 --fc_hw_dim 9_16_26 --expansion 1 \
--single_res --loss Fusion6 --warmup 0.2 --lr_type cosine --strides 5 2 2 2 2 --conv_type conv \
-b 1 --lr 0.0005 --norm none --act swish \
--weight checkpoints/nerv_S.pth --eval_only
Dump predictions with pre-trained model
To evaluate pre-trained model, just add --eval_Only and specify model path with --weight
python train_nerv.py -e 300 --cycles 1 --lower-width 96 --num-blocks 1 --dataset bunny --frame_gap 1 \
--outf bunny_ab --embed 1.25_40 --stem_dim_num 512_1 --reduction 2 --fc_hw_dim 9_16_26 --expansion 1 \
--single_res --loss Fusion6 --warmup 0.2 --lr_type cosine --strides 5 2 2 2 2 --conv_type conv \
-b 1 --lr 0.0005 --norm none --act swish \
--weight checkpoints/nerv_S.pth --eval_only --dump_images
Citation
If you find our work useful in your research, please cite:
@inproceedings{hao2021nerv,
author = {Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser-Nam Lim, Abhinav Shrivastava },
title = {NeRV: Neural Representations for Videos s},
booktitle = {NeurIPS},
year={2021}
}
Contact
If you have any questions, please feel free to email the authors.
420 Jan 05, 2023
493 Jan 04, 2023
602 Jan 07, 2023
305 Dec 21, 2022
1 Jan 12, 2022
66 Dec 04, 2022
3 Oct 18, 2022
0 Aug 05, 2022
2.3k Jan 02, 2023
3 Nov 24, 2022
30 Oct 31, 2022
234 Jan 01, 2023
181 Nov 18, 2022
158 Dec 31, 2022
840 Jan 03, 2023
49 Dec 20, 2022
1 Feb 17, 2022
45 Dec 12, 2022
38 Dec 05, 2022