This is the official code release for the paper Shape and Material Capture at Home

Overview

Shape and Material Capture at Home, CVPR 2021.

Daniel Lichy, Jiaye Wu, Soumyadip Sengupta, David Jacobs

A bare-bones capture setup

Overview

This is the official code release for the paper Shape and Material Capture at Home. The code enables you to reconstruct a 3D mesh and Cook-Torrance BRDF from one or more images captured with a flashlight or camera with flash.

We provide:

  • The trained RecNet model.
  • Code to test on the DiLiGenT dataset.
  • Code to test on our dataset from the paper.
  • Code to test on your own dataset.
  • Code to train a new model, including code for visualization and logging.

Dependencies

This project uses the following dependencies:

  • Python 3.8
  • PyTorch (version = 1.8.1)
  • torchvision
  • numpy
  • scipy
  • opencv
  • OpenEXR (only required for training)

The easiest way to run the code is by creating a virtual environment and installing the dependences with pip e.g.

# Create a new python3.8 environment named py3.8
virtualenv py3.8 -p python3.8

# Activate the created environment
source py3.8/bin/activate

#upgrade pip
pip install --upgrade pip

# To install dependencies 
python -m pip install -r requirements.txt
#or
python -m pip install -r requirements_no_exr.txt

Capturing you own dataset

Multi-image captures

The video below shows how to capture the (up to) six images for you own dataset. Angles are approximate and can be estimated by eye. The camera should be approximately 1 to 4 feet from the object. The flashlight should be far enough from the object such that the entire object is in the illumination cone of the flashlight.

We used this flashlight, but any bright flashlight should work. We used this tripod which comes with a handy remote for iPhone and Android.

Please see the Project Page for a higher resolution version of this video.

Example reconstructions:


Single image captures

Our network also provides state-of-the-art results for reconstructing shape and material from a single flash image.

Examples captured with just an iPhone with flash enabled in a dim room (complete darkness is not needed):


Mask Making

For best performance you should supply a segmentation mask with your image. For our paper we used https://github.com/saic-vul/fbrs_interactive_segmentation which enables mask making with just a few clicks.

Normal prediction results are reasonable without the mask, but integrating normals to a mesh without the mask can be challenging.

Test RecNet on the DiLiGenT dataset

# Download and prepare the DiLiGenT dataset
sh scripts/prepare_diligent_dataset.sh

# Test on 3 DiLiGenT images from the front, front-right, and front-left
# if you only have CPUs remove the --gpu argument
python eval_diligent.py results_path --gpu

# To test on a different subset of DiLiGenT images use the argument --image_nums n1 n2 n3 n4 n5 n6
# where n1 to n6 are the image indices of the right, front-right, front, front-left, left, and above
# images, respectively. For images that are no present set the image number to -1
# e.g to test on only the front image (image number 51) run
python eval_diligent.py results_path --gpu --image_nums -1 -1 51 -1 -1 -1 

Test on our dataset/your own dataset

The easiest way to test on you own dataset and our dataset is to format it as follows:

dataset_dir:

  • sample_name1:
    • 0.ext (right)
    • 1.ext (front-right)
    • 2.ext (front)
    • 3.ext (front-left)
    • 4.ext (left)
    • 5.ext (above)
    • mask.ext
  • sample_name2: (if not all images are present just don't add it to the directory)
    • 2.ext (front)
    • 3.ext (front-left)
  • ...

Where .ext is the image extention e.g. .png, .jpg, .exr

For an example of formating your own dataset please look in data/sample_dataset

Then run:

python eval_standard.py results_path --dataset_root path_to_dataset_dir --gpu

# To test on a sample of our dataset run
python eval_standard.py results_path --dataset_root data/sample_dataset --gpu

Download our real dataset

Coming Soon...

Integrating Normal Maps and Producing a Mesh

We include a script to integrate normals and produce a ply mesh with per vertex albedo and roughness.

After running eval_standard.py or eval_diligent.py there with be a file results_path/images/integration_data.csv Running the following command with produce a ply mesh in results_path/images/sample_name/mesh.ply

python integrate_normals.py results_path/images/integration_data.csv --gpu

This is the most time intensive part of the reconstruction and takes about 3 minutes to run on GPU and 5 minutes on CPU.

Training

To train RecNet from scratch:

python train.py log_dir --dr_dataset_root path_to_dr_dataset --sculpt_dataset_root path_to_sculpture_dataset --gpu

Download the training data

Coming Soon...

FAQ

Q1: What should I do if I have problem running your code?

  • Please create an issue if you encounter errors when trying to run the code. Please also feel free to submit a bug report.

Citation

If you find this code or the provided models useful in your research, please cite it as:

@inproceedings{lichy_2021,
  title={Shape and Material Capture at Home},
  author={Lichy, Daniel and Wu, Jiaye and Sengupta, Soumyadip and Jacobs, David W.},
  booktitle={CVPR},
  year={2021}
}

Acknowledgement

Code used for downloading and loading the DiLiGenT dataset is adapted from https://github.com/guanyingc/SDPS-Net

[CVPR 2020] Transform and Tell: Entity-Aware News Image Captioning

Transform and Tell: Entity-Aware News Image Captioning This repository contains the code to reproduce the results in our CVPR 2020 paper Transform and

Alasdair Tran 85 Dec 13, 2022
This repository contains the exercises and its solution contained in the book "An Introduction to Statistical Learning" in python.

An-Introduction-to-Statistical-Learning This repository contains the exercises and its solution contained in the book An Introduction to Statistical L

2.1k Jan 02, 2023
Adversarial Reweighting for Partial Domain Adaptation

Adversarial Reweighting for Partial Domain Adaptation Code for paper "Xiang Gu, Xi Yu, Yan Yang, Jian Sun, Zongben Xu, Adversarial Reweighting for Par

12 Dec 01, 2022
💡 Type hints for Numpy

Type hints with dynamic checks for Numpy! (❒) Installation pip install nptyping (❒) Usage (❒) NDArray nptyping.NDArray lets you define the shape and

Ramon Hagenaars 377 Dec 28, 2022
基于PaddleOCR搭建的OCR server... 离线部署用

开头说明 DangoOCR 是基于大家的 CPU处理器 来运行的,CPU处理器 的好坏会直接影响其速度, 但不会影响识别的精度 ,目前此版本识别速度可能在 0.5-3秒之间,具体取决于大家机器的配置,可以的话尽量不要在运行时开其他太多东西。需要配合团子翻译器 Ver3.6 及其以上的版本才可以使用!

胖次团子 131 Dec 25, 2022
Source code for Fixed-Point GAN for Cloud Detection

FCD: Fixed-Point GAN for Cloud Detection PyTorch source code of Nyborg & Assent (2020). Abstract The detection of clouds in satellite images is an ess

Joachim Nyborg 8 Dec 22, 2022
Code for the paper: Fighting Fake News: Image Splice Detection via Learned Self-Consistency

Fighting Fake News: Image Splice Detection via Learned Self-Consistency [paper] [website] Minyoung Huh *12, Andrew Liu *1, Andrew Owens1, Alexei A. Ef

minyoung huh (jacob) 174 Dec 09, 2022
deep learning model that learns to code with drawing in the Processing language

sketchnet sketchnet - processing code generator can we teach a computer to draw pictures with code. We use Processing and java/jruby code paired with

41 Dec 12, 2022
My implementation of Fully Convolutional Neural Networks in Keras

Keras-FCN This repository contains my implementation of Fully Convolutional Networks in Keras (Tensorflow backend). Currently, semantic segmentation c

The Duy Nguyen 15 Jan 13, 2020
Cobalt Strike teamserver detection.

Cobalt-Strike-det Cobalt Strike teamserver detection. usage: cobaltstrike_verify.py [-l TARGETS] [-t THREADS] optional arguments: -h, --help show this

TimWhite 17 Sep 27, 2022
D²Conv3D: Dynamic Dilated Convolutions for Object Segmentation in Videos

D²Conv3D: Dynamic Dilated Convolutions for Object Segmentation in Videos This repository contains the implementation for "D²Conv3D: Dynamic Dilated Co

17 Oct 20, 2022
A GOOD REPRESENTATION DETECTS NOISY LABELS

A GOOD REPRESENTATION DETECTS NOISY LABELS This code is a PyTorch implementation of the paper: Prerequisites Python 3.6.9 PyTorch 1.7.1 Torchvision 0.

<a href=[email protected]"> 64 Jan 04, 2023
Official implementations of PSENet, PAN and PAN++.

News (2021/11/03) Paddle implementation of PAN, see Paddle-PANet. Thanks @simplify23. (2021/04/08) PSENet and PAN are included in MMOCR. Introduction

395 Dec 14, 2022
A Python Reconnection Tool for alt:V

altv-reconnect What? It invokes a reconnect in the altV Client Dev Console. You get to determine when your local client should reconnect when developi

8 Jun 30, 2022
Gesture Volume Control v.2

Gesture volume control v.2 In this project I am going to learn how to use Gesture Control to change the volume of a computer. I first look into hand t

Pavel Dat 23 Dec 26, 2022
[CVPR 2022 Oral] EPro-PnP: Generalized End-to-End Probabilistic Perspective-n-Points for Monocular Object Pose Estimation

EPro-PnP EPro-PnP: Generalized End-to-End Probabilistic Perspective-n-Points for Monocular Object Pose Estimation In CVPR 2022 (Oral). [paper] Hanshen

同济大学智能汽车研究所综合感知研究组 ( Comprehensive Perception Research Group under Institute of Intelligent Vehicles, School of Automotive Studies, Tongji University) 842 Jan 04, 2023
Python inverse kinematics for your robot model based on Pinocchio.

Python inverse kinematics for your robot model based on Pinocchio.

Stéphane Caron 50 Dec 22, 2022
Pytorch modules for paralel models with same architecture. Ideal for multi agent-based systems

WideLinears Pytorch parallel Neural Networks A package of pytorch modules for fast paralellization of separate deep neural networks. Ideal for agent-b

1 Dec 17, 2021
Put blind watermark into a text with python

text_blind_watermark Put blind watermark into a text. Can be used in Wechat dingding ... How to Use install pip install text_blind_watermark Alice Pu

郭飞 164 Dec 30, 2022
Repository for benchmarking graph neural networks

Benchmarking Graph Neural Networks Updates Nov 2, 2020 Project based on DGL 0.4.2. See the relevant dependencies defined in the environment yml files

NTU Graph Deep Learning Lab 2k Jan 03, 2023