Volume Rendering of Neural Implicit Surfaces

Project Page | Paper | Data

This repository contains an implementation for the NeurIPS 2021 paper:
Volume Rendering of Neural Implicit Surfaces
Lior Yariv¹, Jiatao Gu², Yoni Kasten¹, Yaron Lipman^1,2
1Weizmann Institute of Science, ²Facebook AI Research

The paper introduce VolSDF: a volume rendering framework for implicit neural surfaces, allowing to learn high fidelity geometry from a sparse set of input images.

Setup

Installation Requirmenets

The code is compatible with python 3.8 and pytorch 1.7. In addition, the following packages are required:
numpy, pyhocon, plotly, scikit-image, trimesh, imageio, opencv, torchvision.

You can create an anaconda environment called volsdf with the required dependencies by running:

conda env create -f environment.yml
conda activate volsdf

Data

We apply our multiview surface reconstruction model to real 2D images from two datasets: DTU and BlendedMVS. The selected scans data evaluated in the paper can be downloaded using:

bash data/download_data.sh 

For more information on the data convention and how to run VolSDF on a new data please have a look at data convention.

Usage

Multiview 3D reconstruction

For training VolSDF run:

cd ./code
python training/exp_runner.py --conf ./confs/dtu.conf --scan_id SCAN_ID

where SCAN_ID is the id of the scene to reconstruct.

To run on the BlendedMVS dataset, which have more complex background, use --conf ./confs/bmvs.conf.

Evaluation

To produce the meshed surface and renderings, run:

cd ./code
python evaluation/eval.py  --conf ./confs/dtu.conf --scan_id SCAN_ID --checkpoint CHECKPOINT [--eval_rendering]

where CHECKPOINT is the epoch you wish to evaluate or 'latest' if you wish to take the most recent epoch. Turning on --eval_rendering will further produce and evaluate PSNR of train image reconstructions.

Citation

If you find our work useful in your research, please consider citing:

@inproceedings{yariv2021volume,
  title={Volume rendering of neural implicit surfaces},
  author={Yariv, Lior and Gu, Jiatao and Kasten, Yoni and Lipman, Yaron},
  booktitle={Thirty-Fifth Conference on Neural Information Processing Systems},
  year={2021}
}