Pip-package for trajectory benchmarking from "Be your own Benchmark: No-Reference Trajectory Metric on Registered Point Clouds", ECMR'21

Related tags

Deep Learningmap-metrics
Overview

Map Metrics for Trajectory Quality

Map metrics toolkit provides a set of metrics to quantitatively evaluate trajectory quality via estimating consistency of the map aggregated from point clouds.

GPS or Motion Capture systems are not always available in perception systems, or their quality is not enough (GPS on small-scale distances) for use as ground truth trajectory. Thus, common full-reference trajectory metrics (APE, RPE, and their modifications) could not be applied to evaluate trajectory quality. When 3D sensing technologies (depth camera, LiDAR) are available on the perception system, one can alternatively assess trajectory quality --- estimate the consistency of the map from registered point clouds via the trajectory.

Documentation: https://map-metrics.readthedocs.io.

Documentation Status Updates

Features

Our toolkit provides implementation of the next metrics:

  • Mean Map Entropy (MME), Mean Plane Variance(MPV) [1] [2]
  • Mutually Orthogonal Metric (MOM) [3] -- has strong correlation with RPE

Citation

If you use this toolkit or MOM-metric results, please, cite our work:

@misc{kornilova2021benchmark,
    title={Be your own Benchmark: No-Reference Trajectory Metric on Registered Point Clouds},
    author={Anastasiia Kornilova and Gonzalo Ferrer},
    year={2021},
    eprint={2106.11351},
    archivePrefix={arXiv},
    primaryClass={cs.RO}
}

Credits

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.

Links

[1] Droeschel, David, Jörg Stückler, and Sven Behnke. "Local multi-resolution representation for 6D motion estimation and mapping with a continuously rotating 3D laser scanner." 2014 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2014.
[2] Razlaw, Jan, et al. "Evaluation of registration methods for sparse 3D laser scans." 2015 European Conference on Mobile Robots (ECMR). IEEE, 2015.
[3] Kornilova, Anastasiia, and Gonzalo Ferrer. "Be your own Benchmark: No-Reference Trajectory Metric on Registered Point Clouds." arXiv preprint arXiv:2106.11351 (2021).
Comments
  • Cross-Platform Wheels

    Cross-Platform Wheels

    Linux and Windows32-64 wheels.

    MacOS is running into from map_metrics import map_metrics E ImportError: dlopen(/Users/runner/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/map_metrics/map_metrics.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libboost_graph-mt-x64.dylib E Referenced from: /Users/runner/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/map_metrics/map_metrics.cpython-36m-darwin.so E Reason: image not found

    opened by achains 1
  • Speed up Github Actions workflow

    Speed up Github Actions workflow

    It takes an hour to build Open3D

    Possible solutions:

    • [ ] Add -j flag
    • [ ] Specify Open3D build options
    • [ ] Store Open3D files in Github Actions Cache
    enhancement 
    opened by achains 1
  • Python version interface improvements

    Python version interface improvements

    Tasks:

    • [x] Break down large functions
    • [x] Break functions into modules
    • [x] Build pip-package, configure CI
    • [x] Add more functions, e.g. I/O processing
    opened by achains 0
  • pip-package. MPV. MME

    pip-package. MPV. MME

    Overview

    Python package wheels with implemented basic metrics (MPV, MME) on C++

    Platform tags

    • manylinux2010_x86_64
    • macosx_10_14_x86_64
    • win32 / win_amd64

    Python version

    • Python >=3.6
    opened by achains 0
  • Cosmetic fixes

    Cosmetic fixes

    What was done:

    • Removed redundant comments
    • Implemented general interface for methods
    • Added missing headers
    • Unsigned long -> int
    • All functions have CamelCase now
    opened by achains 0
  • CMake project structure. Baseline MME and MPV

    CMake project structure. Baseline MME and MPV

    Progress:

    • Configured CMake files
    • Baseline of mme and mpv algorithms

    Notes (Tasks for next PR)

    • Algorithms need to be tested
    • CI should be configured
    opened by achains 0
  • A drawback of mom when dealing drafting walls in indoor envs.

    A drawback of mom when dealing drafting walls in indoor envs.

    • Map Metrics 0.0.1:
    • Python version 3.8:
    • Operating System win10: 1

    Description

    It tired the metirc on a small scale data, likely a room with differential chassis and a RS-16. When the point cloud map consists of multiple wall (actually one) caused by drifting, the mom metric may have a smaller value than a normal map. I think it is caused by the orthogonal walls (both the real one and drifted one) and floor.

    Specifically, Fig.1 is generated by LOAM (without drift), and Fig. 2 is gererated by Lio-sam (with draft).

    Fig.1

    Fig.2

    I also considered to downsample the two maps into the same scale with voxel downsampling. The results are listed below.

    1640311688(1)

    I think this kind of problem is very familiar for indoor, I have seen many drafting wall during daily usage of lidar based slam, without semantic labels, it is very hard for distinguishing them automatically.

    What I Did

    I will try the 0.0.2 version later.

    Paste the command(s) you ran and the output.
If there was a crash, please include the traceback here.
    opened by hahakid 2
  • Add C++ executable with console interface

    Add C++ executable with console interface

    It will be useful for quick testing and performance measurments if library is able to run from command-line interface.

    For example, ./map-metrics --pc="data/kitti_00" --tj="data/Tj_0" --metric="mme" [--"some common hyperparameters"]

    It's build should be disabled by default

    enhancement 
    opened by achains 0
  • Support 10.9 MacOS

    Support 10.9 MacOS

    • Map Metrics version: 0.0.3
    • Python version: >=3.6
    • Operating System: MacOS

    Description

    Pip-package supports MacOS >=10.14 due to C++17 features. We can downgrade to C++14, but a bit later :)

    enhancement 
    opened by achains 0
  • editoring error

    editoring error

    A \delta seems missing in Eq.6 of the paper. Have you tested on all the sequences and other dataset. Recommend to ref LIBRE: The Multiple 3D LiDAR Dataset. VLP64 may have a low precise for long distance measurement. clipboard

    opened by hahakid 1
Owner
Mobile Robotics Lab. at Skoltech
Mobile Robotics Lab. at Skoltech
GitHub Repository
Code for the paper One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation, CVPR 2021.

One Thing One Click One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation (CVPR2021) Code for the paper One Thi

44 Dec 12, 2022
Baseline of DCASE 2020 task 4

Couple Learning for SED This repository provides the data and source code for sound event detection (SED) task. The improvement of the Couple Learning

21 Oct 18, 2022
DVG-Face: Dual Variational Generation for Heterogeneous Face Recognition, TPAMI 2021

DVG-Face: Dual Variational Generation for HFR This repo is a PyTorch implementation of DVG-Face: Dual Variational Generation for Heterogeneous Face Re

52 Dec 30, 2022
Software & Hardware to do multi color printing with Sharpies

3D Print Colorizer is a combination of 3D printed parts and a Cura plugin which allows anyone with an Ender 3 like 3D printer to produce multi colored

343 Jan 06, 2023
EgoNN: Egocentric Neural Network for Point Cloud Based 6DoF Relocalization at the City Scale

EgonNN: Egocentric Neural Network for Point Cloud Based 6DoF Relocalization at the City Scale Paper: EgoNN: Egocentric Neural Network for Point Cloud

19 Sep 20, 2022
Python and C++ implementation of "MarkerPose: Robust real-time planar target tracking for accurate stereo pose estimation". Accepted at LXCV @ CVPR 2021.

MarkerPose: Robust real-time planar target tracking for accurate stereo pose estimation This is a PyTorch and LibTorch implementation of MarkerPose: a

Jhacson Meza 47 Nov 18, 2022
Python scripts for performing lane detection using the LSTR model in ONNX

ONNX LSTR Lane Detection Python scripts for performing lane detection using the Lane Shape Prediction with Transformers (LSTR) model in ONNX. Requirem

Ibai Gorordo 29 Aug 30, 2022
Spatial Transformer Nets in TensorFlow/ TensorLayer

MOVED TO HERE Spatial Transformer Networks Spatial Transformer Networks (STN) is a dynamic mechanism that produces transformations of input images (or

Hao 36 Nov 23, 2022
NeurIPS-2021: Neural Auto-Curricula in Two-Player Zero-Sum Games.

NAC Official PyTorch implementation of NAC from the paper: Neural Auto-Curricula in Two-Player Zero-Sum Games. We release code for: Gradient based ora

Xidong Feng 19 Nov 11, 2022
PointCNN: Convolution On X-Transformed Points (NeurIPS 2018)

PointCNN: Convolution On X-Transformed Points Created by Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Introduction PointCNN

Yangyan Li 1.3k Dec 21, 2022
Magic tool for managing internet connection in local network by @zalexdev

Megacut ✂️ A new powerful Python3 tool for managing internet on a local network Installation git clone https://github.com/stryker-project/megacut cd m

Stryker 12 Dec 15, 2022
Multimodal Co-Attention Transformer (MCAT) for Survival Prediction in Gigapixel Whole Slide Images

Multimodal Co-Attention Transformer (MCAT) for Survival Prediction in Gigapixel Whole Slide Images [ICCV 2021] © Mahmood Lab - This code is made avail

Mahmood Lab @ Harvard/BWH 63 Dec 01, 2022
Easy to use and customizable SOTA Semantic Segmentation models with abundant datasets in PyTorch

Semantic Segmentation Easy to use and customizable SOTA Semantic Segmentation models with abundant datasets in PyTorch Features Applicable to followin

sithu3 530 Jan 05, 2023
Repository for the AugmentedPCA Python package.

Overview This Python package provides implementations of Augmented Principal Component Analysis (AugmentedPCA) - a family of linear factor models that

Billy Carson 6 Dec 07, 2022
[NeurIPS 2019] Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss

Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, Tengyu Ma This is the offi

Kaidi Cao 528 Jan 01, 2023
The self-supervised goal reaching benchmark introduced in Discovering and Achieving Goals via World Models

Lexa-Benchmark Codebase for the self-supervised goal reaching benchmark introduced in 'Discovering and Achieving Goals via World Models'. Setup Create

1 Oct 14, 2021
Bunch of different tools which helps visualizing and annotating images for semantic/instance segmentation tasks

Data Framework for Semantic/Instance Segmentation Bunch of different tools which helps visualizing, transforming and annotating images for semantic/in

Bruno Fernandes Carvalho 5 Dec 21, 2022
:hot_pepper: R²SQL: "Dynamic Hybrid Relation Network for Cross-Domain Context-Dependent Semantic Parsing." (AAAI 2021)

R²SQL The PyTorch implementation of paper Dynamic Hybrid Relation Network for Cross-Domain Context-Dependent Semantic Parsing. (AAAI 2021) Requirement

huybery 60 Dec 31, 2022
Simple renderer for use with MuJoCo (>=2.1.2) Python Bindings.

Viewer for MuJoCo in Python Interactive renderer to use with the official Python bindings for MuJoCo. Starting with version 2.1.2, MuJoCo comes with n

Rohan P. Singh 62 Dec 30, 2022
A Python library that provides a simplified alternative to DBAPI 2

A Python library that provides a simplified alternative to DBAPI 2. It provides a facade in front of DBAPI 2 drivers.

Tony Locke 44 Nov 17, 2021