HDMapNet: A Local Semantic Map Learning and Evaluation Framework

Related tags

Deep LearningHDMapNet
Overview

HDMapNet_devkit

Devkit for HDMapNet.

HDMapNet: A Local Semantic Map Learning and Evaluation Framework

Qi Li, Yue Wang, Yilun Wang, Hang Zhao

[Paper] [Project Page] [5-min video]

Abstract: Estimating local semantics from sensory inputs is a central component for high-definition map constructions in autonomous driving. However, traditional pipelines require a vast amount of human efforts and resources in annotating and maintaining the semantics in the map, which limits its scalability. In this paper, we introduce the problem of local semantic map learning, which dynamically constructs the vectorized semantics based on onboard sensor observations. Meanwhile, we introduce a local semantic map learning method, dubbed HDMapNet. HDMapNet encodes image features from surrounding cameras and/or point clouds from LiDAR, and predicts vectorized map elements in the bird's-eye view. We benchmark HDMapNet on nuScenes dataset and show that in all settings, it performs better than baseline methods. Of note, our fusion-based HDMapNet outperforms existing methods by more than 50% in all metrics. In addition, we develop semantic-level and instance-level metrics to evaluate the map learning performance. Finally, we showcase our method is capable of predicting a locally consistent map. By introducing the method and metrics, we invite the community to study this novel map learning problem. Code and evaluation kit will be released to facilitate future development.

Questions/Requests: Please file an issue or email me at [email protected].

Preparation

  1. Download nuScenes dataset and put it to dataset/ folder.

  2. Install dependencies by running

pip install -r requirement.txt

Vectorization

Run python vis_label.py for demo of vectorized labels. The visualizations are in dataset/nuScenes/samples/GT.

Evaluation

Run python evaluate.py --result_path [submission file] for evaluation. The script accepts vectorized or rasterized maps as input. For vectorized map, We firstly rasterize the vectors to map to do evaluation. For rasterized map, you should make sure the line width=1.

Below is the format for vectorized submission:

-- Whether this submission uses camera data as an input. "use_lidar": -- Whether this submission uses lidar data as an input. "use_radar": -- Whether this submission uses radar data as an input. "use_external": -- Whether this submission uses external data as an input. "vector": true -- Whether this submission uses vector format. }, "results": { sample_token : List[vectorized_line] -- Maps each sample_token to a list of vectorized lines. } } vectorized_line { "pts": List[ ] -- Ordered points to define the vectorized line. "pts_num": , -- Number of points in this line. "type": <0, 1, 2> -- Type of the line: 0: ped; 1: divider; 2: boundary "confidence_level": -- Confidence level for prediction (used by Average Precision) }">
vectorized_submission {
    "meta": {
        "use_camera":   
          
             -- Whether this submission uses camera data as an input.
        "use_lidar":    
           
              -- Whether this submission uses lidar data as an input.
        "use_radar":    
            
               -- Whether this submission uses radar data as an input.
        "use_external": 
             
                -- Whether this submission uses external data as an input.
        "vector":        true   -- Whether this submission uses vector format.
    },
    "results": {
        sample_token 
              
               : List[vectorized_line] -- Maps each sample_token to a list of vectorized lines. } } vectorized_line { "pts": List[
               
                ] -- Ordered points to define the vectorized line. "pts_num": 
                
                 , -- Number of points in this line. "type": <0, 1, 2> -- Type of the line: 0: ped; 1: divider; 2: boundary "confidence_level": 
                 
                   -- Confidence level for prediction (used by Average Precision) } 
                 
                
               
              
             
            
           
          

For rasterized submission, the format is:

-- Whether this submission uses camera data as an input. "use_lidar": -- Whether this submission uses lidar data as an input. "use_radar": -- Whether this submission uses radar data as an input. "use_external": -- Whether this submission uses external data as an input. "vector": false -- Whether this submission uses vector format. }, "results": { sample_token : { -- Maps each sample_token to a list of vectorized lines. "map": [ ], -- Raster map of prediction (C=0: ped; 1: divider 2: boundary). The value indicates the line idx (start from 1). "confidence_level": Array[float], -- confidence_level[i] stands for confidence level for i^th line (start from 1). } } }">
rasterized_submisson {
    "meta": {
        "use_camera":   
        
           -- Whether this submission uses camera data as an input.
        "use_lidar":    
         
            -- Whether this submission uses lidar data as an input.
        "use_radar":    
          
             -- Whether this submission uses radar data as an input.
        "use_external": 
           
              -- Whether this submission uses external data as an input.
        "vector":       false   -- Whether this submission uses vector format.
    },
    "results": {
        sample_token 
            
             : { -- Maps each sample_token to a list of vectorized lines. "map": [
             
              ], -- Raster map of prediction (C=0: ped; 1: divider 2: boundary). The value indicates the line idx (start from 1). "confidence_level": Array[float], -- confidence_level[i] stands for confidence level for i^th line (start from 1). } } } 
             
            
           
          
         
        

Run python export_to_json.py to get a demo of vectorized submission. Run python export_to_json.py --raster for rasterized submission.

Citation

If you found this useful in your research, please consider citing

@misc{li2021hdmapnet,
      title={HDMapNet: A Local Semantic Map Learning and Evaluation Framework}, 
      author={Qi Li and Yue Wang and Yilun Wang and Hang Zhao},
      year={2021},
      eprint={2107.06307},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}
Owner
Tsinghua MARS Lab
MARS Lab at IIIS, Tsinghua University
Tsinghua MARS Lab
Trainable PyTorch reproduction of AlphaFold 2

OpenFold A faithful PyTorch reproduction of DeepMind's AlphaFold 2. Features OpenFold carefully reproduces (almost) all of the features of the origina

AQ Laboratory 1.7k Dec 29, 2022
HCQ: Hybrid Contrastive Quantization for Efficient Cross-View Video Retrieval

HCQ: Hybrid Contrastive Quantization for Efficient Cross-View Video Retrieval [toc] 1. Introduction This repository provides the code for our paper at

13 Dec 08, 2022
Unofficial PyTorch implementation of MobileViT.

MobileViT Overview This is a PyTorch implementation of MobileViT specified in "MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Tr

Chin-Hsuan Wu 348 Dec 23, 2022
[ICCV2021] Learning to Track Objects from Unlabeled Videos

Unsupervised Single Object Tracking (USOT) 🌿 Learning to Track Objects from Unlabeled Videos Jilai Zheng, Chao Ma, Houwen Peng and Xiaokang Yang 2021

53 Dec 28, 2022
KIDA: Knowledge Inheritance in Data Aggregation

KIDA: Knowledge Inheritance in Data Aggregation This project releases our 1st place solution on NeurIPS2021 ML4CO Dual Task. Slide and model weights a

24 Sep 08, 2022
Official PyTorch code for Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021)

Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021) This repository is the official P

Jingyun Liang 159 Dec 30, 2022
WarpRNNT loss ported in Numba CPU/CUDA for Pytorch

RNNT loss in Pytorch - Numba JIT compiled (warprnnt_numba) Warp RNN Transducer Loss for ASR in Pytorch, ported from HawkAaron/warp-transducer and a re

Somshubra Majumdar 15 Oct 22, 2022
Laser device for neutralizing - mosquitoes, weeds and pests

Laser device for neutralizing - mosquitoes, weeds and pests (in progress) Here I will post information for creating a laser device. A warning!! How It

Ildaron 1k Jan 02, 2023
Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers

Segmentation Transformer Implementation of Segmentation Transformer in PyTorch, a new model to achieve SOTA in semantic segmentation while using trans

Abhay Gupta 161 Dec 08, 2022
Official Code for AdvRush: Searching for Adversarially Robust Neural Architectures (ICCV '21)

AdvRush Official Code for AdvRush: Searching for Adversarially Robust Neural Architectures (ICCV '21) Environmental Set-up Python == 3.6.12, PyTorch =

11 Dec 10, 2022
Point Cloud Registration using Representative Overlapping Points.

Point Cloud Registration using Representative Overlapping Points (ROPNet) Abstract 3D point cloud registration is a fundamental task in robotics and c

ZhuLifa 36 Dec 16, 2022
Adversarial examples to the new ConvNeXt architecture

Adversarial examples to the new ConvNeXt architecture To get adversarial examples to the ConvNeXt architecture, run the Colab: https://github.com/stan

Stanislav Fort 19 Sep 18, 2022
🏆 The 1st Place Submission to AICity Challenge 2021 Natural Language-Based Vehicle Retrieval Track (Alibaba-UTS submission)

AI City 2021: Connecting Language and Vision for Natural Language-Based Vehicle Retrieval 🏆 The 1st Place Submission to AICity Challenge 2021 Natural

82 Dec 29, 2022
Autoencoder - Reducing the Dimensionality of Data with Neural Network

autoencoder Implementation of the Reducing the Dimensionality of Data with Neural Network – G. E. Hinton and R. R. Salakhutdinov paper. Notes Aim to m

Jordan Burgess 13 Nov 17, 2022
Implemenets the Contourlet-CNN as described in C-CNN: Contourlet Convolutional Neural Networks, using PyTorch

C-CNN: Contourlet Convolutional Neural Networks This repo implemenets the Contourlet-CNN as described in C-CNN: Contourlet Convolutional Neural Networ

Goh Kun Shun (KHUN) 10 Nov 03, 2022
COVID-Net Open Source Initiative

The COVID-Net models provided here are intended to be used as reference models that can be built upon and enhanced as new data becomes available

Linda Wang 1.1k Dec 26, 2022
GANTheftAuto is a fork of the Nvidia's GameGAN

Description GANTheftAuto is a fork of the Nvidia's GameGAN, which is research focused on emulating dynamic game environments. The early research done

Harrison 801 Dec 27, 2022
Learning to Prompt for Continual Learning

Learning to Prompt for Continual Learning (L2P) Official Jax Implementation L2P is a novel continual learning technique which learns to dynamically pr

Google Research 207 Jan 06, 2023
Open source Python module for computer vision

About PCV PCV is a pure Python library for computer vision based on the book "Programming Computer Vision with Python" by Jan Erik Solem. More details

Jan Erik Solem 1.9k Jan 06, 2023
The pytorch implementation of DG-Font: Deformable Generative Networks for Unsupervised Font Generation

DG-Font: Deformable Generative Networks for Unsupervised Font Generation The source code for 'DG-Font: Deformable Generative Networks for Unsupervised

130 Dec 05, 2022