Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis

Last update: Dec 27, 2022

Related tags

Overview

Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis

This repository contains the official implementation of DietNeRF, a system that reconstructs 3D scenes from a few posed photos.

Setup

We use the following folder structure:

dietnerf/
  logs/ (images, videos, checkpoints)
  data/
    nerf_synthetic/
  configs/ (run configuration files)
CLIP/ (Fork of OpenAI's clip repository with a wrapper)

Create conda environment:

conda create -n dietnerf python=3.9
conda activate dietnerf

Set up requirements and our fork of CLIP:

pip install -r requirements.txt
cd CLIP
pip install -e .

wandb login

Experiments on the Realistic Synthetic dataset

Realistic Synthetic experiments are implemented in the ./dietnerf subdirectory.

You need to download datasets from NeRF's Google Drive folder. The dataset was used in the original NeRF paper by Mildenhall et al. For example,

mkdir dietnerf/logs/ dietnerf/data/
cd dietnerf/data
pip install gdown
gdown --id 18JxhpWD-4ZmuFKLzKlAw-w5PpzZxXOcG -O nerf_synthetic.zip
unzip nerf_synthetic.zip
rm -r __MACOSX

Then, shrink images to 400x400:

python dietnerf/scripts/bulk_shrink_images.py "dietnerf/data/nerf_synthetic/*/*/*.png" dietnerf/data/nerf_synthetic_400_rgb/ True

These images are used for FID/KID computation. The dietnerf/run_nerf.py training and evaluation code automatically shrinks images with the --half_res argument.

Each experiment has a config file stored in dietnerf/configs/. Scripts in dietnerf/scripts/ can be run to train and evaluate models. Run these scripts from ./dietnerf. The scripts assume you are running one script at a time on a server with 8 NVIDIA GPUs.

cd dietnerf
export WANDB_ENTITY=
   
    

# NeRF baselines
sh scripts/run_synthetic_nerf_100v.sh
sh scripts/run_synthetic_nerf_8v.sh
sh scripts/run_synthetic_simplified_nerf_8v.sh

# DietNeRF with 8 observed views
sh scripts/run_synthetic_dietnerf_8v.sh
sh scripts/run_synthetic_dietnerf_ft_8v.sh

# NeRF and DietNeRF with partial observability
sh scripts/run_synthetic_unseen_side_14v.sh

Experiments on the DTU dataset

Coming soon. Our paper also fine-tunes pixelNeRF on DTU scenes for 1-shot view synthesis.

Citation and acknowledgements

If DietNeRF is relevant to your project, please cite our associated paper:

@InProceedings{Jain_2021_ICCV,
    author    = {Jain, Ajay and Tancik, Matthew and Abbeel, Pieter},
    title     = {Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2021},
    pages     = {5885-5894}
}

This code is based on Yen-Chen Lin's PyTorch implementation of NeRF and the official pixelNeRF code.

Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis

Related tags

Overview

Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis

Setup

Experiments on the Realistic Synthetic dataset

Experiments on the DTU dataset

Citation and acknowledgements

Owner

Ajay Jain

[NeurIPS 2021] The PyTorch implementation of paper "Self-Supervised Learning Disentangled Group Representation as Feature"

A PyTorch implementation of "Signed Graph Convolutional Network" (ICDM 2018).

f-BRS: Rethinking Backpropagating Refinement for Interactive Segmentation

Official Repo for Ground-aware Monocular 3D Object Detection for Autonomous Driving

PyTorch implementation for our NeurIPS 2021 Spotlight paper "Long Short-Term Transformer for Online Action Detection".

Pytorch implementation of CVPR2020 paper “VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation”

POCO: Point Convolution for Surface Reconstruction

Official repository for Automated Learning Rate Scheduler for Large-Batch Training (8th ICML Workshop on AutoML)

AQP is a modular pipeline built to enable the comparison and testing of different quality metric configurations.

This repository contains the code for the paper "PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization"

Explainable Medical ImageSegmentation via GenerativeAdversarial Networks andLayer-wise Relevance Propagation

Bringing Computer Vision and Flutter together , to build an awesome app !!

Python program that works as a contact list

An unofficial personal implementation of UM-Adapt, specifically to tackle joint estimation of panoptic segmentation and depth prediction for autonomous driving datasets.

PyTorch CZSL framework containing GQA, the open-world setting, and the CGE and CompCos methods.

Code for "AutoMTL: A Programming Framework for Automated Multi-Task Learning"

Wide Residual Networks (WideResNets) in PyTorch

Diffusion Probabilistic Models for 3D Point Cloud Generation (CVPR 2021)

Combining Latent Space and Structured Kernels for Bayesian Optimization over Combinatorial Spaces

This is an official implementation for "ResT: An Efficient Transformer for Visual Recognition".