RGB-D Local Implicit Function for Depth Completion of Transparent Objects

Overview

RGB-D Local Implicit Function for Depth Completion of Transparent Objects

[Project Page] [Paper]

Overview

This repository maintains the official implementation of our CVPR 2021 paper:

RGB-D Local Implicit Function for Depth Completion of Transparent Objects

By Luyang Zhu, Arsalan Mousavian, Yu Xiang, Hammad Mazhar, Jozef van Eenbergen, Shoubhik Debnath, Dieter Fox

Requirements

The code has been tested on the following system:

  • Ubuntu 18.04
  • Nvidia GPU (4 Tesla V100 32GB GPUs) and CUDA 10.2
  • python 3.7
  • pytorch 1.6.0

Installation

Docker (Recommended)

We provide a Dockerfile for building a container to run our code. More details about GPU accelerated Docker containers can be found here.

Local Installation

We recommend creating a new conda environment for a clean installation of the dependencies.

conda create --name lidf python=3.7
conda activate lidf

Make sure CUDA 10.2 is your default cuda. If your CUDA 10.2 is installed in /usr/local/cuda-10.2, add the following lines to your ~/.bashrc and run source ~/.bashrc:

export PATH=$PATH:/usr/local/cuda-10.2/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-10.2/lib64
export CPATH=$CPATH:/usr/local/cuda-10.2/include

Install libopenexr-dev

sudo apt-get update && sudo apt-get install libopenexr-dev

Install dependencies, we use ${REPO_ROOT_DIR} to represent the working directory of this repo.

cd ${REPO_ROOT_DIR}
conda install pytorch==1.6.0 torchvision==0.7.0 cudatoolkit=10.2 -c pytorch
pip install -r requirements.txt

Dataset Preparation

ClearGrasp Dataset

ClearGrasp can be downloaded at their official website (Both training and testing dataset are needed). After you download zip files and unzip them on your local machine, the folder structure should be like

${DATASET_ROOT_DIR}
├── cleargrasp
│   ├── cleargrasp-dataset-train
│   ├── cleargrasp-dataset-test-val

Omniverse Object Dataset

Omniverse Object Dataset can be downloaded here. After you download zip files and unzip them on your local machine, the folder structure should be like

${DATASET_ROOT_DIR}
├── omniverse
│   ├── train
│   │	├── 20200904
│   │	├── 20200910

Soft link dataset

cd ${REPO_ROOT_DIR}
ln -s ${DATASET_ROOT_DIR}/cleargrasp datasets/cleargrasp
ln -s ${DATASET_ROOT_DIR}/omniverse datasets/omniverse

Testing

We provide pretrained checkpoints at the Google Drive. After you download the file, please unzip and copy the checkpoints folder under ${REPO_ROOT_DIR}.

Change the following line in ${REPO_ROOT_DIR}/src/experiments/implicit_depth/run.sh:

# To test first stage model (LIDF), use the following line
cfg_paths=experiments/implicit_depth/test_lidf.yaml
# To test second stage model (refinement model), use the following line
cfg_paths=experiments/implicit_depth/test_refine.yaml

After that, run the testing code:

cd src
bash experiments/implicit_depth/run.sh

Training

First stage model (LIDF)

Change the following line in ${REPO_ROOT_DIR}/src/experiments/implicit_depth/run.sh:

cfg_paths=experiments/implicit_depth/train_lidf.yaml

After that, run the training code:

cd src
bash experiments/implicit_depth/run.sh

Second stage model (refinement model)

In ${REPO_ROOT_DIR}/src/experiments/implicit_depth/train_refine.yaml, set lidf_ckpt_path to the path of the best checkpoint in the first stage training. Change the following line in ${REPO_ROOT_DIR}/src/experiments/implicit_depth/run.sh:

cfg_paths=experiments/implicit_depth/train_refine.yaml

After that, run the training code:

cd src
bash experiments/implicit_depth/run.sh

Second stage model (refinement model) with hard negative mining

In ${REPO_ROOT_DIR}/src/experiments/implicit_depth/train_refine_hardneg.yaml, set lidf_ckpt_path to the path of the best checkpoint in the first stage training, set checkpoint_path to the path of the best checkpoint in the second stage training. Change the following line in ${REPO_ROOT_DIR}/src/experiments/implicit_depth/run.sh:

cfg_paths=experiments/implicit_depth/train_refine_hardneg.yaml

After that, run the training code:

cd src
bash experiments/implicit_depth/run.sh

License

This work is licensed under NVIDIA Source Code License - Non-commercial.

Citation

If you use this code for your research, please citing our work:

@inproceedings{zhu2021rgbd,
author    = {Luyang Zhu and Arsalan Mousavian and Yu Xiang and Hammad Mazhar and Jozef van Eenbergen and Shoubhik Debnath and Dieter Fox},
title     = {RGB-D Local Implicit Function for Depth Completion of Transparent Objects},
booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
year      = {2021}
}
Owner
NVIDIA Research Projects
NVIDIA Research Projects
Behavioral "black-box" testing for recommender systems

RecList RecList Free software: MIT license Documentation: https://reclist.readthedocs.io. Overview RecList is an open source library providing behavio

Jacopo Tagliabue 375 Dec 30, 2022
Diverse Branch Block: Building a Convolution as an Inception-like Unit

Diverse Branch Block: Building a Convolution as an Inception-like Unit (PyTorch) (CVPR-2021) DBB is a powerful ConvNet building block to replace regul

253 Dec 24, 2022
Official implementation of "Watermarking Images in Self-Supervised Latent-Spaces"

🔍 Watermarking Images in Self-Supervised Latent-Spaces PyTorch implementation and pretrained models for the paper. For details, see Watermarking Imag

Meta Research 32 Dec 13, 2022
Implements Stacked-RNN in numpy and torch with manual forward and backward functions

Recurrent Neural Networks Implements simple recurrent network and a stacked recurrent network in numpy and torch respectively. Both flavours implement

Vishal R 1 Nov 16, 2021
PAIRED in PyTorch 🔥

PAIRED This codebase provides a PyTorch implementation of Protagonist Antagonist Induced Regret Environment Design (PAIRED), which was first introduce

UCL DARK Lab 46 Dec 12, 2022
Automates Machine Learning Pipeline with Feature Engineering and Hyper-Parameters Tuning :rocket:

MLJAR Automated Machine Learning Documentation: https://supervised.mljar.com/ Source Code: https://github.com/mljar/mljar-supervised Table of Contents

MLJAR 2.4k Dec 31, 2022
The Rich Get Richer: Disparate Impact of Semi-Supervised Learning

The Rich Get Richer: Disparate Impact of Semi-Supervised Learning Preprocess file of the dataset used in implicit sub-populations: (Demographic groups

<a href=[email protected]"> 4 Oct 14, 2022
HyperLib: Deep learning in the Hyperbolic space

HyperLib: Deep learning in the Hyperbolic space Background This library implements common Neural Network components in the hypberbolic space (using th

105 Dec 25, 2022
A python toolbox for predictive uncertainty quantification, calibration, metrics, and visualization

Website, Tutorials, and Docs    Uncertainty Toolbox A python toolbox for predictive uncertainty quantification, calibration, metrics, and visualizatio

Uncertainty Toolbox 1.4k Dec 28, 2022
Computational modelling of ray propagation through optical elements using the principles of geometric optics (Ray Tracer)

Computational modelling of ray propagation through optical elements using the principles of geometric optics (Ray Tracer) Introduction By applying the

Son Gyo Jung 1 Jul 09, 2022
PyTorch implementation of Super SloMo by Jiang et al.

Super-SloMo PyTorch implementation of "Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation" by Jiang H., Sun

Avinash Paliwal 2.9k Jan 03, 2023
Global-Local Path Networks for Monocular Depth Estimation with Vertical CutDepth [Paper]

Global-Local Path Networks for Monocular Depth Estimation with Vertical CutDepth [Paper] Downloads [Downloads] Trained ckpt files for NYU Depth V2 and

98 Jan 01, 2023
Implementation of Fast Transformer in Pytorch

Fast Transformer - Pytorch Implementation of Fast Transformer in Pytorch. This only work as an encoder. Yannic video AI Epiphany Install $ pip install

Phil Wang 167 Dec 27, 2022
这是一个deeplabv3-plus-pytorch的源码,可以用于训练自己的模型。

DeepLabv3+:Encoder-Decoder with Atrous Separable Convolution语义分割模型在Pytorch当中的实现 目录 性能情况 Performance 所需环境 Environment 注意事项 Attention 文件下载 Download 训练步骤

Bubbliiiing 350 Dec 28, 2022
Rename Images with Auto Generated Neural Image Captions

Recaption Images with Generated Neural Image Caption Example Usage: Commandline: Recaption all images from folder /home/feng/Downloads/images to folde

feng wang 3 May 01, 2022
Code, Data and Demo for Paper: Controllable Generation from Pre-trained Language Models via Inverse Prompting

InversePrompting Paper: Controllable Generation from Pre-trained Language Models via Inverse Prompting Code: The code is provided in the "chinese_ip"

THUDM 101 Dec 16, 2022
Impelmentation for paper Feature Generation and Hypothesis Verification for Reliable Face Anti-Spoofing

FGHV Impelmentation for paper Feature Generation and Hypothesis Verification for Reliable Face Anti-Spoofing Requirements Python 3.6 Pytorch 1.5.0 Cud

5 Jun 02, 2022
The authors' official PyTorch SigWGAN implementation

The authors' official PyTorch SigWGAN implementation This repository is the official implementation of [Sig-Wasserstein GANs for Time Series Generatio

9 Jun 16, 2022
Multi-Task Learning as a Bargaining Game

Nash-MTL Official implementation of "Multi-Task Learning as a Bargaining Game". Setup environment conda create -n nashmtl python=3.9.7 conda activate

Aviv Navon 87 Dec 26, 2022
TorchDistiller - a collection of the open source pytorch code for knowledge distillation, especially for the perception tasks, including semantic segmentation, depth estimation, object detection and instance segmentation.

This project is a collection of the open source pytorch code for knowledge distillation, especially for the perception tasks, including semantic segmentation, depth estimation, object detection and i

yifan liu 147 Dec 03, 2022