Symmetry and Uncertainty-Aware Object SLAM for 6DoF Object Pose Estimation

Related tags

Deep Learningsuo_slam
Overview

SUO-SLAM

This repository hosts the code for our CVPR 2022 paper "Symmetry and Uncertainty-Aware Object SLAM for 6DoF Object Pose Estimation". ArXiv link.

Citation

If you use any part of this repository in an academic work, please cite our paper as:

@inproceedings{Merrill2022CVPR,
  Title      = {Symmetry and Uncertainty-Aware Object SLAM for 6DoF Object Pose Estimation},
  Author     = {Nathaniel Merrill and Yuliang Guo and Xingxing Zuo and Xinyu Huang and Stefan Leutenegger and Xi Peng and Liu Ren and Guoquan Huang},
  Booktitle  = {2022 Conference on Computer Vision and Pattern Recognition (CVPR)},
  Year       = {2022},
  Address    = {New Orleans, USA},
  Month      = jun,
}

Installation

Click for details... This codebase was tested on Ubuntu 18.04. To use the BOP rendering (i.e. for keypoint labeling) install
sudo apt install libfreetype6-dev libglfw3

You will also need a python environment that contains the required packages. To see what packages we used, check out the list of requirements in requirements.txt. They can be installed via pip install -r requirements.txt

Preparing Data

Click for details...

Datasets

To be able to run the training and testing (i.e. single view or with SLAM), first decide on a place to download the data to. The disk will need a few hundred GB of space for all the data (at least 150GB for download and more to extract). All of our code expects the data to be in a local directory ./data, but you can of course symlink this to another location (perhaps with more disk space). So, first of all, in the root of this repo run

$ mkdir data

or to symlink to an external location

$ ln -s /path/to/drive/with/space/ ./data

You can pick and choose what data you want to download (for example just T-LESS or YCBV). Note that all YCBV and TLESS downloads have our keypoint labels packaged along with the data. Download the following google drive links into ./data and extract them.

When all is said and done, the tree should look like this

$ cd ./data && tree --filelimit 3
.
├── bop_datasets
│   ├── tless 
│   └── ycbv 
├── saved_detections
└── VOCdevkit
    └── VOC2012

Pre-trained models

You can download the pretrained models anywhere, but I like to keep them in the results directory that is written to during training.

Training

Click for details...

First set the default arguments in ./lib/args.py for your username if desired, then execute

$ ./train.py

with the appropriate arguments for your filesystem. You can also run

$ ./train.py -h

for a full list of arguments and their meaning. Some important args are batch_size, which is the number of images loaded for each training batch. Note that there may be a variable number of objects in each image, and the objects are all stacked together into one big batch to run the network -- so the actual batch size being run might be multiple times batch_size. In order to keep batch_size reasonably large, we provide another arg called truncate_obj, which, as the help says, truncates the object batches to this number if it exceeds it. We recommend that you start with a large batch size so that you can find out the maximum truncate_obj for you GPUs, then reduce the batch size until there are little to no warnings about too many objects being truncated.

Evaluation

Click for details...

Before you can evaluate in a single-view or SLAM fashion, you will need to build the thirdparty libraries for PnP and graph optimization. First make sure that you have CERES solver installed. The run

$ ./build_thirdparty.sh

Reproducing Results

To reproduce the results of the paper with the pretrained models, check out the scripts under the scripts directory:

eval_all_tless.sh  eval_all_ycbv.sh  make_video.sh

These will reproduce most of the results in the paper as well as any video clips you want. You may have to change the first few lines of each script. Note that these examples can also show you the proper arguments if you want to run from command line alone.

Note that for the T-LESS dataset, we use the thirdparty BOP toolkit to get the VSD error recall, which will show up in the final terminal output as "Mean object recall" among other numbers.

Labeling

Click for details...

Overview

We manually label keypoints on the CAD model to enable some keypoints with semantic meaning. For the full list of keypoint meanings, see the specific README

We provide our landmark labeling tool. Check out the script manual_keypoints.py. This same script can be used to make a visualization of the keypoints as shown below with the --viz option.

The script will show a panel of the same object but oriented slightly differently. The idea is that you pick the same keypoint multiple times to ensure correctness and to get a better label by averaging multiple samples.

The script will also print the following directions to follow in the terminal.

============= Welcome ===============
Select the keypoints with a left click!
Use the "wasd" to turn the objects.
Press "i" to zoom in and "o" to zoom out.
Make sure that the keypoint colors match between all views.
Messed up? Just press 'u' to undo.
Press "Enter" to finish and save the keypoints
Press "Esc" to just quit

Once you have pressed "enter", you will get to an inspection pane.

Where the unscaled mean keypoints are on the left, and the ones scaled by covariance is on the left, where the ellipses are the Gaussian 3-sigma projected onto the image. If the covariance is too large, or the mean is out of place, then you may have messed up. Again, the program will print out these directions to terminal:

Inspect the results!
Use the "wasd" to turn the object.
Press "i" to zoom in and "o" to zoom out.
Press "Esc" to go back, "Enter" to accept (saving keypoints and viewpoint for vizualization).
Please pick a point on the object!

So if you are done, and the result looks good, then press "Enter", if not then "Esc" to go back. Make sure also that when you are done, you rotate and scale the object into the best "view pose" (with the front facing the camera, and top facing up), as this pose is used by both the above vizualization and the actual training code for determining the best symmetry to pick for an initial detection.

Labeling Tips

Even though there are 8 panels, you don't need to fill out all 8. Each keypoint just needs at least 3 samples to sample the covariance.

We recommend that you label the same keypoint (say keypoint i) on all the object renderings first, then go to the inspection panel at the end of this each time so that you can easily undo a mistake for keypoint i with the "u" key and not lose any work. Otherwise, if you label each object rendering completely, then you may have to undo a lot of labelings that were not mistakes.

Also, if there is an object that you want to label a void in the CAD model, like the top center of the bowl, then you can use the multiple samples to your advantage, and choose samples that will average to the desired result, since the labels are required to land on the actual CAD model in the labeling tool.

<\details>

Owner
Robot Perception & Navigation Group (RPNG)
Research on robot sensing, estimation, localization, mapping, perception, and planning
Robot Perception & Navigation Group (RPNG)
Self-supervised Label Augmentation via Input Transformations (ICML 2020)

Self-supervised Label Augmentation via Input Transformations Authors: Hankook Lee, Sung Ju Hwang, Jinwoo Shin (KAIST) Accepted to ICML 2020 Install de

hankook 96 Dec 29, 2022
Keras implementation of AdaBound

AdaBound for Keras Keras port of AdaBound Optimizer for PyTorch, from the paper Adaptive Gradient Methods with Dynamic Bound of Learning Rate. Usage A

Somshubra Majumdar 132 Sep 23, 2022
Fast, accurate and reliable software for algebraic CT reconstruction

KCT CBCT Fast, accurate and reliable software for algebraic CT reconstruction. This set of software tools includes OpenCL implementation of modern CT

Vojtěch Kulvait 4 Dec 14, 2022
EGNN - Implementation of E(n)-Equivariant Graph Neural Networks, in Pytorch

EGNN - Pytorch Implementation of E(n)-Equivariant Graph Neural Networks, in Pytorch. May be eventually used for Alphafold2 replication. This

Phil Wang 259 Jan 04, 2023
The official repository for Deep Image Matting with Flexible Guidance Input

FGI-Matting The official repository for Deep Image Matting with Flexible Guidance Input. Paper: https://arxiv.org/abs/2110.10898 Requirements easydict

Hang Cheng 51 Nov 10, 2022
A PyTorch implementation of Learning to learn by gradient descent by gradient descent

Intro PyTorch implementation of Learning to learn by gradient descent by gradient descent. Run python main.py TODO Initial implementation Toy data LST

Ilya Kostrikov 300 Dec 11, 2022
Second Order Optimization and Curvature Estimation with K-FAC in JAX.

KFAC-JAX - Second Order Optimization with Approximate Curvature in JAX Installation | Quickstart | Documentation | Examples | Citing KFAC-JAX KFAC-JAX

DeepMind 90 Dec 22, 2022
Fast image augmentation library and easy to use wrapper around other libraries. Documentation: https://albumentations.ai/docs/ Paper about library: https://www.mdpi.com/2078-2489/11/2/125

Albumentations Albumentations is a Python library for image augmentation. Image augmentation is used in deep learning and computer vision tasks to inc

11.4k Jan 09, 2023
Replication Package for AequeVox:Automated Fariness Testing for Speech Recognition Systems

AequeVox Replication Package for AequeVox:Automated Fariness Testing for Speech Recognition Systems README under development. Python Packages Required

Sai Sathiesh 2 Aug 28, 2022
Little tool in python to watch anime from the terminal (the better way to watch anime)

ani-cli Script working again :), thanks to the fork by Dink4n for the alternative approach to by pass the captcha on gogoanime A cli to browse and wat

Harshith 4.5k Dec 31, 2022
Code for "Modeling Indirect Illumination for Inverse Rendering", CVPR 2022

Modeling Indirect Illumination for Inverse Rendering Project Page | Paper | Data Preparation Set up the python environment conda create -n invrender p

ZJU3DV 116 Jan 03, 2023
Multi Agent Path Finding Algorithms

MATP-solver Simulator collision check path step random initial states or given states Traditional method Seperate A* algorithem Confict-based Search S

30 Dec 12, 2022
Semantic Bottleneck Scene Generation

SB-GAN Semantic Bottleneck Scene Generation Coupling the high-fidelity generation capabilities of label-conditional image synthesis methods with the f

Samaneh Azadi 41 Nov 28, 2022
This is the code for Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning

This is the code for Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning It includes /bert, which is the original BERT repos

Mitchell Gordon 11 Nov 15, 2022
[peer review] An Arbitrary Scale Super-Resolution Approach for 3D MR Images using Implicit Neural Representation

ArSSR This repository is the pytorch implementation of our manuscript "An Arbitrary Scale Super-Resolution Approach for 3-Dimensional Magnetic Resonan

Qing Wu 19 Dec 12, 2022
Some code of the implements of Geological Modeling Using 3D Pixel-Adaptive and Deformable Convolutional Neural Network

3D-GMPDCNN Geological Modeling Using 3D Pixel-Adaptive and Deformable Convolutional Neural Network PyTorch implementation of "Geological Modeling Usin

5 Nov 21, 2022
Repository for the paper : Meta-FDMixup: Cross-Domain Few-Shot Learning Guided byLabeled Target Data

1 Meta-FDMIxup Repository for the paper : Meta-FDMixup: Cross-Domain Few-Shot Learning Guided byLabeled Target Data. (ACM MM 2021) paper News! the rep

Fu Yuqian 44 Nov 18, 2022
Official PyTorch implementation of CAPTRA: CAtegory-level Pose Tracking for Rigid and Articulated Objects from Point Clouds

CAPTRA: CAtegory-level Pose Tracking for Rigid and Articulated Objects from Point Clouds Introduction This is the official PyTorch implementation of o

Yijia Weng 96 Dec 07, 2022
A curated list of resources for Image and Video Deblurring

A curated list of resources for Image and Video Deblurring

Subeesh Vasu 1.7k Jan 01, 2023
DeepDiffusion: Unsupervised Learning of Retrieval-adapted Representations via Diffusion-based Ranking on Latent Feature Manifold

DeepDiffusion Introduction This repository provides the code of the DeepDiffusion algorithm for unsupervised learning of retrieval-adapted representat

4 Nov 15, 2022