GeoTransformer - Geometric Transformer for Fast and Robust Point Cloud Registration

Overview

Geometric Transformer for Fast and Robust Point Cloud Registration

PyTorch implementation of the paper:

Geometric Transformer for Fast and Robust Point Cloud Registration.

Zheng Qin, Hao Yu, Changjian Wang, Yulan Guo, Yuxing Peng, and Kai Xu

Introduction

We study the problem of extracting accurate correspondences for point cloud registration. Recent keypoint-free methods bypass the detection of repeatable keypoints which is difficult in low-overlap scenarios, showing great potential in registration. They seek correspondences over downsampled superpoints, which are then propagated to dense points. Superpoints are matched based on whether their neighboring patches overlap. Such sparse and loose matching requires contextual features capturing the geometric structure of the point clouds. We propose Geometric Transformer to learn geometric feature for robust superpoint matching. It encodes pair-wise distances and triplet-wise angles, making it robust in low-overlap cases and invariant to rigid transformation. The simplistic design attains surprisingly high matching accuracy such that no RANSAC is required in the estimation of alignment transformation, leading to $100$ times acceleration. Our method improves the inlier ratio by $17% \sim 30%$ and the registration recall by over $7%$ on the challenging 3DLoMatch benchmark. Code will be released for paper reproduction.

News

Installation

Please use the following command for installation.

# It is recommended to create a new environment
conda create -n geotransformer python==3.8
conda activate geotransformer

# [Optional] If you are using CUDA 11.0 or newer, please install `torch==1.7.1+cu110`
pip install torch==1.7.1+cu110 -f https://download.pytorch.org/whl/torch_stable.html

# Install packages and other dependencies
python setup.py build develop

# Compile c++ wrappers
cd geotransformer/cpp_wrappers
sh ./compile_wrappers.sh

Code has been tested with Ubuntu 20.04, GCC 9.3.0, Python 3.8, PyTorch 1.7.1, CUDA 11.1 and cuDNN 8.1.0.

Data preparation

We provide code for training and testing on 3DMatch.

The dataset can be download from PREDATOR. The data should be organized as follows:

--data--3DMatch--metadata
              |--data--train--7-scenes-chess--cloud_bin_0.pth
                    |      |               |--...
                    |      |--...
                    |--test--7-scenes-redkitchen--cloud_bin_0.pth
                          |                    |--...
                          |--...

Training

The code for GeoTransformer is in experiments/geotransformer.3dmatch. Use the following command for training.

CUDA_VISIBLE_DEVICES=0 python trainval.py
# use "--snapshot=path/to/snapshot" to resume training.

Testing

Use the following command for testing.

# 3DMatch
CUDA_VISIBLE_DEVICES=0 ./eval.sh EPOCH 3DMatch
# 3DLoMatch
CUDA_VISIBLE_DEVICES=0 ./eval.sh EPOCH 3DLoMatch

EPOCH is the epoch id.

We also provide pretrained weights in weights, use the following command to test the pretrained weights.

CUDA_VISIBLE_DEVICES=0 python test.py --snapshot=../../weights/geotransformer-3dmatch.pth.tar --benchmark=3DMatch
CUDA_VISIBLE_DEVICES=0 python eval.py --run_matching --run_registration --benchmark=3DMatch

Replace 3DMatch with 3DLoMatch to evaluate on 3DLoMatch.

Results

Benchmark FMR IR RR
3DMatch 97.7 70.3 91.5
3DLoMatch 88.1 43.3 74.0
Comments
  • Test and visualize on two given point clouds

    Test and visualize on two given point clouds

    Hi, thanks for sharing your work. Is it easy to give some guidance or do you have any tutorial to test/evaluate the pre-trained models on two individual given point clouds?

    opened by ttsesm 10
  • Question about kitti result reproduce.

    Question about kitti result reproduce.

    Hi ,thanks for your amazing work.

    However , I have met a problem of kitti dataset result reproduce. I followed your code to train and test on kitti dataset , but my result seems always lower than the result in your paper. I can achieve Registration recall of 99.8 and RTE 6.8 same as yours ,but my result of RRE is about 0.33 which is much higher than 0.24 in your paper.

    Can you give me some idea of how to reproduce the same result ?

    opened by vision507 9
  • How to draw the point cloud like this ?

    How to draw the point cloud like this ?

    Hi , thanks for your great work. And I would like to know how can I get this kind of visualization result in your paper. 屏幕截图 2022-08-04 081115 When I use open3d to draw the point clouds , it seems very ugly, so I'm wondering how can I get your visualization result. It would be very grateful.

    opened by vision507 9
  • How to group data sets after downloading

    How to group data sets after downloading

    Hello, author: I ran the python files in the Kitti dataset and ModelNet dataset folders according to your requirements, but they did not work and were not classified into the ones you described. Is there a specific method? thank you!

    opened by yangsihai97 9
  • Some questions about downsampling of point clouds

    Some questions about downsampling of point clouds

    Hi there,

    Really appreciate releasing your amazing work! I'm quite new to point clouds and I have some questions regarding downsampling of the point clouds.

    Take 3DMatch as an example. The original raw dataset downloaded from PREDATOR is quite dense, and is preprocessed so that each grid size is 2.5cm. In your work, you further subsample the point clouds using KPConv (the first level downsampled points). My first question is what's the purpose of subsampling? Would a denser point cloud be more informative for registration?

    Also, is there a specific reason you downsample the points further for dense point correspondences as I believe the preprocessed input data is already of density 2.5cm?

    Last, is there a quick way to visualize the registration result for the 3DMatch benchmark? By running eval.py and test.py it seems to only produce the quantitative metrics.

    Thanks for your patience and reply in advance!

    opened by jinhong-ni 5
  • 3DMatch dataset

    3DMatch dataset

    Hi,

    I have a question regarding the D.2. section from the Supplementary material.

    Could you please share which 3DMatch files you use for the evaluation?

    • Do you use those with overlap > 30%?
    • Do you use consecutive pairs of point cloud indices (the original 3DMatch filters the pairs (i,j) with i+1 < j)?

    You mention that you use the files from PREDATOR, but PREDATOR has more pairs than the original 3DMatch dataset. Can you please elaborate?

    Thank you in advance for you response. Hopefully we can clear all this confusion of the registration pairs used in the datasets. Every paper seems to do it a bit differently.

    opened by DavidBoja 4
  • why need gt transoform in demo.py

    why need gt transoform in demo.py

    in line 74 in experiments/geotransformer.3dmatch.stage4.gse.k3.max.oacl.stage2.sinkhorn/model.py , transform = data_dict['transform'].detach(), and in line 110 you use gt transform in get_node_correspondences. however If i don't know how the transform matrix of src to ref, how can I use demo.py

    opened by MaybeOjbk 4
  • The different RR in test.py and eval.py

    The different RR in test.py and eval.py

    Hello, thank you for you amazing work again. When I evaluated on 3DMatch, I found the RR reported by test.py and eval.py are very different, what makes this difference?

    opened by Hui-design 4
  • Question about radius search

    Question about radius search

    First of all, thanks for your excellent work! Radius search in KPConv is quiet slow for large scale point clouds and it usually takes seconds in my own experiments. Is there a more efficient way for radius search, such as a version that can run on GPU. Hope to get your answer!

    opened by Fleurrr 4
  • Questions about RANSAC

    Questions about RANSAC

    Dear authors,

    Thanks for open-sourcing your amazing work. I'm experimenting with RANSAC and I noticed some unexpected results compared to the paper.

    What I've done is first change the number of iterations in RANSAC to 50k. The evaluation runs extremely slowly (approx. 16 hrs in contrast to about 1 hr for local to global), and I only got performance similar to local to global. I also tried to change the number of points in RANSAC to a larger number (by default it is set to 3), such as 250, 500, 1000, etc. I did not achieve a significant gain in performance, even the recall collapses to zero if the number of points is set too big (like 5k). This is contradicting to what's reported in the paper, as the performance is reported to keep improving as the number of samples increases.

    I would be super grateful if you could guide me through what I've done wrong here. Thanks in advance for your patience and help!

    opened by jinhong-ni 4
  • RuntimeError: Error compiling objects for extension

    RuntimeError: Error compiling objects for extension

    Hello, sir. The question is about creating the code runtime environment. After reinstalling CUDA and downloading torch, I executed 'python setup.py build develop' and it still report the error, 'RuntimeError: Error compiling objects for extension'.

    Traceback (most recent call last): File "setup.py", line 5, in setup( File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_init_.py", line 87, in setup return distutils.core.setup(**attrs) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\core.py", line 148, in setup return run_commands(dist) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\core.py", line 163, in run_commands dist.run_commands() File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\dist.py", line 967, in run_commands self.run_command(cmd) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools\dist.py", line 1214, in run_command super().run_command(command) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\dist.py", line 986, in run_command cmd_obj.run() File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\command\build.py", line 135, in run self.run_command(cmd_name) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\cmd.py", line 313, in run_command self.distribution.run_command(command) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools\dist.py", line 1214, in run_command super().run_command(command) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\dist.py", line 986, in run_command cmd_obj.run() File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools\command\build_ext.py", line 79, in run _build_ext.run(self) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\command\build_ext.py", line 339, in run self.build_extensions() File "F:\Anaconda3\envs\geotransformer\lib\site-packages\torch\utils\cpp_extension.py", line 670, in build_extensions build_ext.build_extensions(self) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\command\build_ext.py", line 448, in build_extensions self._build_extensions_serial() File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\command\build_ext.py", line 473, in _build_extensions_serial self.build_extension(ext) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools\command\build_ext.py", line 202, in build_extension _build_ext.build_extension(self, ext) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\command\build_ext.py", line 528, in build_extension objects = self.compiler.compile(sources, File "F:\Anaconda3\envs\geotransformer\lib\site-packages\torch\utils\cpp_extension.py", line 643, in win_wrap_ninja_compile _write_ninja_file_and_compile_objects( File "F:\Anaconda3\envs\geotransformer\lib\site-packages\torch\utils\cpp_extension.py", line 1250, in _write_ninja_file_and_compile_objects _run_ninja_build( File "F:\Anaconda3\envs\geotransformer\lib\site-packages\torch\utils\cpp_extension.py", line 1555, in _run_ninja_build raise RuntimeError(message) from e RuntimeError: Error compiling objects for extension

    Is this because I use win10? Or other issues caused the error?

    opened by ChaunceyQ 4
  • The training model is not effective

    The training model is not effective

    Thank you for your excellent work. When I use the LGR training model without changing the code, the result on 3DMatch (about 91.0%) was still quite different from the model you provided (92.5%)

    opened by awdx551 3
Owner
Zheng Qin
computer vision, deep learning
Zheng Qin
SAMO: Streaming Architecture Mapping Optimisation

SAMO: Streaming Architecture Mapping Optimiser The SAMO framework provides a method of optimising the mapping of a Convolutional Neural Network model

Alexander Montgomerie-Corcoran 20 Dec 10, 2022
Minimal But Practical Image Classifier Pipline Using Pytorch, Finetune on ResNet18, Got 99% Accuracy on Own Small Datasets.

PyTorch Image Classifier Updates As for many users request, I released a new version of standared pytorch immage classification example at here: http:

JinTian 106 Nov 06, 2022
PhysCap: Physically Plausible Monocular 3D Motion Capture in Real Time

PhysCap: Physically Plausible Monocular 3D Motion Capture in Real Time The implementation is based on SIGGRAPH Aisa'20. Dependencies Python 3.7 Ubuntu

soratobtai 124 Dec 08, 2022
Official Repo for ICCV2021 Paper: Learning to Regress Bodies from Images using Differentiable Semantic Rendering

[ICCV2021] Learning to Regress Bodies from Images using Differentiable Semantic Rendering Getting Started DSR has been implemented and tested on Ubunt

Sai Kumar Dwivedi 83 Nov 27, 2022
This code is a toolbox that uses Torch library for training and evaluating the ERFNet architecture for semantic segmentation.

ERFNet This code is a toolbox that uses Torch library for training and evaluating the ERFNet architecture for semantic segmentation. NEW!! New PyTorch

Edu 104 Jan 05, 2023
Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more

Apache MXNet (incubating) for Deep Learning Apache MXNet is a deep learning framework designed for both efficiency and flexibility. It allows you to m

The Apache Software Foundation 20.2k Jan 05, 2023
object detection; robust detection; ACM MM21 grand challenge; Security AI Challenger Phase VII

赛题背景 在商品知识产权领域,知识产权体现为在线商品的设计和品牌。不幸的是,在每一天,存在着非法商户通过一些对抗手段干扰商标识别来逃避侵权,这带来了很高的知识产权风险和财务损失。为了促进先进的多媒体人工智能技术的发展,以保护企业来之不易的创作和想法免受恶意使用和剽窃,因此提出了鲁棒性标识检测挑战赛

65 Dec 22, 2022
Finding Biological Plausibility for Adversarially Robust Features via Metameric Tasks

Adversarially-Robust-Periphery Code + Data from the paper "Finding Biological Plausibility for Adversarially Robust Features via Metameric Tasks" by A

Anne Harrington 2 Feb 07, 2022
A PaddlePaddle version of Neural Renderer, refer to its PyTorch version

Neural 3D Mesh Renderer in PadddlePaddle A PaddlePaddle version of Neural Renderer, refer to its PyTorch version Install Run: pip install neural-rende

AgentMaker 13 Jul 12, 2022
Dynamic Capacity Networks using Tensorflow

Dynamic Capacity Networks using Tensorflow Dynamic Capacity Networks (DCN; http://arxiv.org/abs/1511.07838) implementation using Tensorflow. DCN reduc

Taeksoo Kim 8 Feb 23, 2021
Contains supplementary materials for reproduce results in HMC divergence time estimation manuscript

Scalable Bayesian divergence time estimation with ratio transformations This repository contains the instructions and files to reproduce the analyses

Suchard Research Group 1 Sep 21, 2022
Codes for SIGIR'22 Paper 'On-Device Next-Item Recommendation with Self-Supervised Knowledge Distillation'

OD-Rec Codes for SIGIR'22 Paper 'On-Device Next-Item Recommendation with Self-Supervised Knowledge Distillation' Paper, saved teacher models and Andro

Xin Xia 11 Nov 22, 2022
The Instructed Glacier Model (IGM)

The Instructed Glacier Model (IGM) Overview The Instructed Glacier Model (IGM) simulates the ice dynamics, surface mass balance, and its coupling thro

27 Dec 16, 2022
PyTorch implementation of VAGAN: Visual Feature Attribution Using Wasserstein GANs

Prototypical Networks for Few shot Learning in PyTorch Simple alternative Implementation of Prototypical Networks for Few Shot Learning (paper, code)

Orobix 93 Aug 17, 2022
A medical imaging framework for Pytorch

Welcome to MedicalTorch MedicalTorch is an open-source framework for PyTorch, implementing an extensive set of loaders, pre-processors and datasets fo

Christian S. Perone 799 Jan 03, 2023
Flickr-Faces-HQ (FFHQ) is a high-quality image dataset of human faces, originally created as a benchmark for generative adversarial networks (GAN)

Flickr-Faces-HQ Dataset (FFHQ) Flickr-Faces-HQ (FFHQ) is a high-quality image dataset of human faces, originally created as a benchmark for generative

NVIDIA Research Projects 2.9k Dec 28, 2022
PaddleRobotics is an open-source algorithm library for robots based on Paddle, including open-source parts such as human-robot interaction, complex motion control, environment perception, SLAM positioning, and navigation.

简体中文 | English PaddleRobotics paddleRobotics是基于paddle的机器人开源算法库集,包括人机交互、复杂运动控制、环境感知、slam定位导航等开源算法部分。 人机交互 主动多模交互技术TFVT-HRI 主动多模交互技术是通过视觉、语音、触摸传感器等输入机器人

185 Dec 26, 2022
Understanding the Generalization Benefit of Model Invariance from a Data Perspective

Understanding the Generalization Benefit of Model Invariance from a Data Perspective This is the code for our NeurIPS2021 paper "Understanding the Gen

1 Jan 15, 2022
Code for the paper "Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks"

ON-LSTM This repository contains the code used for word-level language model and unsupervised parsing experiments in Ordered Neurons: Integrating Tree

Yikang Shen 572 Nov 21, 2022
A Python library that provides a simplified alternative to DBAPI 2

A Python library that provides a simplified alternative to DBAPI 2. It provides a facade in front of DBAPI 2 drivers.

Tony Locke 44 Nov 17, 2021