POCO: Point Convolution for Surface Reconstruction

Related tags

Deep LearningPOCO
Overview

POCO: Point Convolution for Surface Reconstruction

by: Alexandre Boulch and Renaud Marlet

Abstract

Implicit neural networks have been successfully used for surface reconstruction from point clouds. However, many of them face scalability issues as they encode the isosurface function of a whole object or scene into a single latent vector. To overcome this limitation, a few approaches infer latent vectors on a coarse regular 3D grid or on 3D patches, and interpolate them to answer occupancy queries. In doing so, they loose the direct connection with the input points sampled on the surface of objects, and they attach information uniformly in space rather than where it matters the most, i.e., near the surface. Besides, relying on fixed patch sizes may require discretization tuning. To address these issues, we propose to use point cloud convolutions and compute latent vectors at each input point. We then perform a learning-based interpolation on nearest neighbors using inferred weights. Experiments on both object and scene datasets show that our approach significantly outperforms other methods on most classical metrics, producing finer details and better reconstructing thinner volumes.

Citation

TODO

Dependencies

Our code rely on Pytorch Geometric, which should be installed. Then, run:

python setup.py build_ext --inplace

Data

ShapeNet (Occupancy Network pre-processing)

We use the ShapeNet dataset as pre-processed by Occupancy Networks. Please refer to original repository for downloading the data.

It should be placed in the folder data/ShapeNet.

SyntheticRooms (Occupancy Network pre-processing)

We use the SyntheticRooms dataset as created by Occupancy Networks. Please refer to original repository for downloading the data.

It should be placed in the folder data/synthetic_room_dataset.

ABC / RealWorld / Famous and Thingi10k

They should be placed in the folders: data/3d_shapes_abc, data/3d_shapes_abc_training, data/3d_shapes_famous, data/3d_shapes_real_world and data/3d_shapes_thingi10k.

SceneNet

Download the SceneNet dataset.

Watertight models

We use the code from https://github.com/hjwdzh/Manifold for watertight model creation. Please download and compile it.

To create the watertight models, create a destination directory data/SceneNet, copy-paste the python script datasets/scenenet_watertight.py into the directory and run it with python scenenet_watertight.py. You will need to modify the raw_data_dir and the manifold_code_dir to match your data and code locations.

Creation of the point clouds

You can use the script datasets/scenenet_sample.py. You will also need to modify the paths.

Precomputed meshes and point clouds

For easy use of the code, we provide precomputed meshes, and point clouds SceneNet20, SceneNet100, SceneNet500 and SceneNet1000.

Location

The watertight models should be in data/SceneNet.

The points are located in data/SceneNet20, data/SceneNet100, data/SceneNet1000 and data/SceneNet500.

Training

# train on ShapeNet with 3k points, noise and without normals 
python train.py --config configs/config_shapenet.yaml 

# train on ShapeNet with 10k points, no noise and normals
python train.py --config configs/config_shapenet.yaml --normals True --random_noise 0 --experiment_name Normals

# train on ABC with 3k points
python train.py --config configs/config_abc.yaml --experiment_name 3k

# train on ABC with 10k points
python train.py --config configs/config_abc.yaml --experiment_name 10k --manifold_points 10000 --training_batch_size 8

Generation

ShapeNet

python generate.py --config results/ShapeNet_None_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --gen_resolution_global 128

SyntheticRooms

python generate.py --config results/SyntheticRooms_None_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --gen_resolution_global 256 --num_mesh 1

ABC, RealWorld, Famous and Thingi10k

# Model trained with 3k points
python generate.py --config results/ABC_3k_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --dataset_name DATASET_NAME --dataset_root data/DATASET_DIR --gen_resolution_global 256

python generate.py --config results/ABC_3k_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --dataset_name DATASET_NAME --dataset_root data/DATASET_DIR --gen_resolution_global 256 --manifold_points -1 --gen_subsample_manifold 3000 --gen_subsample_manifold_iter 10 --gen_descriptor gen_sub3k_iter10

# Model trained with 10k points
python generate.py --config results/ABC_10k_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --dataset_name DATASET_NAME --dataset_root data/DATASET_DIR --gen_resolution_global 256

python generate.py --config results/ABC_10k_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --dataset_name DATASET_NAME --dataset_root data/DATASET_DIR --gen_resolution_global 256 --manifold_points -1 --noise 0.0 --gen_subsample_manifold 10000 --gen_subsample_manifold_iter 10 --gen_descriptor gen_sub3k_iter10

For faster generation, one would want to use a lower marching cubes parameter --gen_resolution_global, e.g. 128.

DATASET_NAME should be replaced by ABCTest, ABCTestNoiseFree, ABCTestExtraNoise, RealWorld, FamousTest, FamousTestNoiseFree, FamousTestExtraNoisy, FamousTestSparse, FamousTestDense, Thingi10kTest, Thingi10kTestNoiseFree, Thingi10kTestExtraNoisy, Thingi10kTestSparse, Thingi10kTestDense

DATASET_DIR should be replaced by 3d_shapes_abc, 3d_shapes_famous, 3d_shapes_real_world, 3d_shapes_thingi10k

SceneNet

python generate.py --config results/ShapeNet_Normals_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --gen_autoscale True --gen_autoscale_target 0.01 --gen_resolution_metric 0.01 --dataset_name DATASET_NAME --dataset_root data/DATASET_NAME --manifold_points -1

DATASET_NAME should be replaced by SceneNet20, SceneNet100, SceneNet500, SceneNet1000.

The --gen_autoscale_target parameter is the average distance between a point and its nearest neighbor in the training set, 0.01 for ShapeNet train set with 3000 points.

Evaluation

ShapeNet

python eval_meshes.py --gendir results/ShapeNet_None_FKAConv_InterpAttentionKHeadsNet_None/gen_ShapeNet_test_3000/ --meshdir meshes --dataset ShapeNet --split test --gtdir data/ShapeNet

SyntheticRooms

python eval_meshes.py --gendir results/SyntheticRooms_None_FKAConv_InterpAttentionKHeadsNet_None/gen_SyntheticRooms_test_10000/ --meshdir meshes --dataset SyntheticRooms --split test --gtdir data/synthetic_room_dataset

SceneNet

python eval_scenenet.py --gendir results/ShapeNet_Normals_FKAConv_InterpAttentionKHeadsNet_None/gen_SceneNet20_test_allPts/ --meshdir meshes --gtdir data/SceneNet

ABC, Famous and Thingi10k

For evaluation on the ABCTest please run:

python eval_point2surf_datasets.py --gendir results/ABC_3k_FKAConv_InterpAttentionKHeadsNet_None/gen_ABCTest_test_3000 --meshdir meshes/04_pts/ --gtdir data/3d_shapes_abc/abc/

You can adapt the paths to evaluate on the different datasets and noise variants.

Pretrained models

We provide pre-trained models for FKAConv backbone.

ShapeNet 3k, noise, no normals

ShapeNet 3k, no noise, normals

SyntheticRooms 10k

ABC 3k

ABC 10k

Configuration

The code was used with Ubuntu, Python 3.7.10, Cuda 11.1 and Pytorch 1.8.1

Comments
  • Package Versions

    Package Versions

    Hi,

    I tried to reproduce your results, but I ran into a possible version mismatch between Pytorch and Pytorch_geometric.

    I created my environment with the following commands:

    conda create --name poco python=3.7.10
conda install pytorch==1.8.1 torchvision==0.9.1 torchaudio==0.8.1 cudatoolkit=11.1 -c pytorch -c conda-forge
conda install -c conda-forge cython
conda install -c conda-forge tqdm 
conda install -c conda-forge scikit-image 
conda install -c open3d-admin open3d 
conda install -c conda-forge scikit-learn 
conda install -c conda-forge pyyaml 
conda install -c conda-forge addict 
conda install -c conda-forge pandas 
conda install -c conda-forge plyfile 
conda install -c conda-forge pytorch_geometric

    Compilation with python setup.py build_ext --inplace seems to work but python generate.py --config results/ABC_10k_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --dataset_name DATASET_NAME --dataset_root data/3d_shapes_abc/abc/ --gen_resolution_global 256 results in OSError: /home/perler/miniconda3/envs/poco/lib/python3.7/site-packages/torch_sparse/_version.so: undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE

    Installed versions are:

    (poco) [email protected]:~/repos/poco$ conda list pytorch
# packages in environment at /home/perler/miniconda3/envs/poco:
#
# Name                    Version                   Build  Channel
pytorch                   1.8.1           py3.7_cuda11.1_cudnn8.0.5_0    pytorch
pytorch-cpu               1.1.0               py3.7_cpu_0    pytorch
pytorch_geometric         2.0.3              pyh6c4a22f_0    conda-forge
pytorch_sparse            0.6.4            py37hcae2be3_0    conda-forge

    Again, the CPU-version... but that's a different issue.

    AFAIK, they added sparse tensors only recently to Pytorch, so the installed Pytorch-geometric version might be too new. Which version of Pytorch-geometric do I need?

    Can you please create a requiremtents.txt and/or environment.yaml?

    opened by ErlerPhilipp 13
  • Cython related error when building extensions

    Cython related error when building extensions

    Hi, I completely followed the instructions, using cuda11.1 and Ubuntu, but still having this issue gcc: error: /usr/POCO/build/temp.linux-x86_64-3.7/eval/src/utils/libkdtree/pykdtree/kdtree.o: No such file or directory gcc: error: /usr/POCO/build/temp.linux-x86_64-3.7/eval/src/utils/libkdtree/pykdtree/_kdtree_core.o: No such file or directory It's asking to link .o files before it is generated. But looks like required Cython 0.27.3 is not compatible with Python 3.7.10, which is required by this project. I'm wondering does anyone have a working combination of the version of the packages? Thank you!

    opened by zhaoyuanyuan2011 5
  • Download ABC / RealWorld / Famous and Thingi10k data

    Download ABC / RealWorld / Famous and Thingi10k data

    Hello,

    Thanks for the excellent paper and for releasing the code.

    I'm wondering where I can download the ABC, RealWorld, Famous, and Thingi10k data?

    Best and have a nice day, Mulin

    opened by MulinYu 2
  • Training on my own dataset

    Training on my own dataset

    I met some problem when training on my own dataset, which is processed as ShapeNet, containing pointclouds with normals, SDF points and occ values. However, there is an error when I try to train on it.

    File "/media/yangxilab/DiskB/sanren/3Drecon/POCOnasa/networks/backbone/fkaconv_network.py", line 193, in forward x0 = self.activation(self.bn0(self.cv0(x, pos, pos, data["ids00"]))) KeyError: 'ids00'

    I'm confusing that what data["ids00"] in x0 = self.activation(self.bn0(self.cv0(x, pos, pos, data["ids00"]))) refers to? And similarily, there are many unkown data like data["ids10"]. How could I get this data? Thanks in advance!

    opened by Green89757 2
  • Failed to run

    Failed to run "python setup.py build_ext --inplace"

    Hello,

    After installing Pytorch and Pytorch Geometric, running python setup.py build_ext --inplace gives the following error:

    gcc: error: /my/computer/path/to/POCO/build/temp.linux-x86_64-3.7/eval/src/utils/libkdtree/pykdtree/_kdtree_core.o: No such file or directory
error: command 'gcc' failed with exit status 1

    Any suggestions would be appreciated.

    opened by TixiaoShan 2
  • Bug during generating

    Bug during generating

    Hello again,

    When I try to generate the shapenet results with the command:

    python generate.py --config results/ShapeNet_Normals_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --gen_resolution_global 128

    I got this error:

    **loading CUDA OK INFO:root:Creating the network INFO:root:InterpNet - Simple - K=64 INFO:root:Network -- backbone -- 12783956 parameters INFO:root:Network -- projection -- 6498 parameters INFO:root:Network -- Number of parameters 12790454 INFO:root:Getting the dataset INFO:root:Normals as features INFO:root:Dataset - ShapeNet- None INFO:root:Dataset - len 8751 INFO:root:InterpNet - Simple - K=64 INFO:root:Network -- backbone -- 12783956 parameters INFO:root:Network -- projection -- 6498 parameters

    0%| | 0/8751 [00:00<?, ?it/s] 0%| | 0/8751 [00:04<?, ?it/s] Traceback (most recent call last): File "generate.py", line 594, in main(config) File "generate.py", line 539, in main step=step File "generate.py", line 170, in export_mesh_and_refine_vertices_region_growing_v2 mesh = o3d.geometry.TriangleMesh(o3d_verts, o3d_faces) TypeError: init(): incompatible constructor arguments. The following argument types are supported: 1. open3d.open3d.geometry.TriangleMesh() 2. open3d.open3d.geometry.TriangleMesh(arg0: open3d.open3d.geometry.TriangleMesh)

    Invoked with: std::vectorEigen::Vector3d with 133788 elements. Use numpy.asarray() to access data., std::vectorEigen::Vector3i with 267574 elements. Use numpy.asarray() to access data.

    Did you forget to #include <pybind11/stl.h>? Or <pybind11/complex.h>, <pybind11/functional.h>, <pybind11/chrono.h>, etc. Some automatic conversions are optional and require extra headers to be included when compiling your pybind11 module. **

    Thanks in advance and have a nice day. Best. Mulin

    opened by MulinYu 1
  • It takes one hour to generate the mesh from 16k point clouds in ScanNet dataset

    It takes one hour to generate the mesh from 16k point clouds in ScanNet dataset

    Dear author,

    Thanks for sharing your interesting work!

    I use the provided script to generate the mesh from the ScanNet dataset, and I found that it takes a long time (e.g, nearly 1 hour from 16K point clouds input). I am wondering if it is a normal situation. BTW, I just use the following script to generate mesh; python generate.py --config results/ShapeNet_Normals_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --gen_autoscale True --gen_autoscale_target 0.01 --gen_resolution_metric 0.01 --dataset_name SceneNet20 --dataset_root data/SceneNet20 --manifold_points -1

    Looking forward to your reply. Many thanks in advance.

    Best, Runsong

    opened by Runsong123 0
  • RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when running the code

    RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when running the code

    Hi! Thanks for your interesting and enlightening work on point cloud reconstruction tasks, and we are trying to reproduce your work. However, we encounter an error when running your code: Capture I am wondering how to fix this bug? Thanks for your timely response.

    opened by CUHKWilliam 3
Owner
valeo.ai
We are an international team based in Paris, conducting AI research for Valeo automotive applications, in collaboration with world-class academics.
valeo.ai
GitHub Repository
FedTorch is an open-source Python package for distributed and federated training of machine learning models using PyTorch distributed API

FedTorch is a generic repository for benchmarking different federated and distributed learning algorithms using PyTorch Distributed API.

Machine Learning and Optimization Lab @PennState 136 Dec 23, 2022
Research into Forex price prediction from price history using Deep Sequence Modeling with Stacked LSTMs.

Forex Data Prediction via Recurrent Neural Network Deep Sequence Modeling Research Paper Our research paper can be viewed here Installation Clone the

Alex Taradachuk 2 Aug 07, 2022
A symbolic-model-guided fuzzer for TLS

tlspuffin TLS Protocol Under FuzzINg A symbolic-model-guided fuzzer for TLS Master Thesis | Thesis Presentation | Documentation Disclaimer: The term "

69 Dec 20, 2022
This is a clean and robust Pytorch implementation of DQN and Double DQN.

DQN/DDQN-Pytorch This is a clean and robust Pytorch implementation of DQN and Double DQN. Here is the training curve: All the experiments are trained

XinJingHao 15 Dec 27, 2022
Code for the paper "Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks"

ON-LSTM This repository contains the code used for word-level language model and unsupervised parsing experiments in Ordered Neurons: Integrating Tree

Yikang Shen 572 Nov 21, 2022
Huawei Hackathon 2021 - Sweden (Stockholm)

huawei-hackathon-2021 Contributors DrakeAxelrod Challenge Requirements: python=3.8.10 Standard libraries (no importing) Important factors: Data depend

Drake Axelrod 32 Nov 08, 2022
atmaCup #11 の Public 4th / Pricvate 5th Solution のリポジトリです。

#11 atmaCup 2021-07-09 ~ 2020-07-21 に行われた #11 [初心者歓迎! / 画像編] atmaCup のリポジトリです。結果は Public 4th / Private 5th でした。 フレームワークは PyTorch で、実装は pytorch-image-m

Tawara 12 Apr 07, 2022
SatelliteNeRF - PyTorch-based Neural Radiance Fields adapted to satellite domain

SatelliteNeRF PyTorch-based Neural Radiance Fields adapted to satellite domain.

Kai Zhang 46 Nov 20, 2022
Geometric Vector Perceptrons --- a rotation-equivariant GNN for learning from biomolecular structure

Geometric Vector Perceptron Implementation of equivariant GVP-GNNs as described in Learning from Protein Structure with Geometric Vector Perceptrons b

Dror Lab 142 Dec 29, 2022
Local trajectory planner based on a multilayer graph framework for autonomous race vehicles.

Graph-Based Local Trajectory Planner The graph-based local trajectory planner is python-based and comes with open interfaces as well as debug, visuali

TUM - Institute of Automotive Technology 160 Jan 04, 2023
Semantic Image Synthesis with SPADE

Semantic Image Synthesis with SPADE New implementation available at imaginaire repository We have a reimplementation of the SPADE method that is more

NVIDIA Research Projects 7.3k Jan 07, 2023
Fuzzing the Kernel Using Unicornafl and AFL++

Unicorefuzz Fuzzing the Kernel using UnicornAFL and AFL++. For details, skim through the WOOT paper or watch this talk at CCCamp19. Is it any good? ye

Security in Telecommunications 283 Dec 26, 2022
Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks

Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks This repository contains a TensorFlow implementation of "

Jingwei Zheng 5 Jan 08, 2023
PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Created by Charles R. Qi, Hao Su, Kaichun Mo, Leonidas J. Guibas from Sta

Charles R. Qi 4k Dec 30, 2022
Training Structured Neural Networks Through Manifold Identification and Variance Reduction

Training Structured Neural Networks Through Manifold Identification and Variance Reduction This repository is a pytorch implementation of the Regulari

0 Dec 23, 2021
The code for Expectation-Maximization Attention Networks for Semantic Segmentation (ICCV'2019 Oral)

EMANet News The bug in loading the pretrained model is now fixed. I have updated the .pth. To use it, download it again. EMANet-101 gets 80.99 on the

Xia Li 李夏 663 Nov 30, 2022
Create Own QR code with Python

Create-Own-QR-code Create Own QR code with Python SO guys in here, you have to install pyqrcode 2. open CMD and type python -m pip install pyqrcode

JehanKandy 10 Jul 13, 2022
An official reimplementation of the method described in the INTERSPEECH 2021 paper - Speech Resynthesis from Discrete Disentangled Self-Supervised Representations.

Speech Resynthesis from Discrete Disentangled Self-Supervised Representations Implementation of the method described in the Speech Resynthesis from Di

Facebook Research 253 Jan 06, 2023
Conditional Generative Adversarial Networks (CGAN) for Mobility Data Fusion

This code implements the paper, Kim et al. (2021). Imputing Qualitative Attributes for Trip Chains Extracted from Smart Card Data Using a Conditional Generative Adversarial Network. Transportation Re

Eui-Jin Kim 2 Feb 03, 2022
Long Expressive Memory (LEM)

Long Expressive Memory for Sequence Modeling This repository contains the implementation to reproduce the numerical experiments of the paper Long Expr

Konstantin Rusch 47 Dec 17, 2022