PyTorch implementation of 'Gen-LaneNet: a generalized and scalable approach for 3D lane detection'

Overview

(pytorch) Gen-LaneNet: a generalized and scalable approach for 3D lane detection

Introduction

This is a pytorch implementation of Gen-LaneNet, which predicts 3D lanes from a single image. Specifically, Gen-LaneNet is a unified network solution that solves image encoding, spatial transform of features and 3D lane prediction simultaneously. The method refers to the ECCV 2020 paper:

'Gen-LaneNet: a generalized and scalable approach for 3D lane detection', Y Guo, etal. ECCV 2020. [eccv][arxiv]

Key features:

  • A geometry-guided lane anchor representation generalizable to novel scenes.

  • A scalable two-stage framework that decouples the learning of image segmentation subnetwork and geometry encoding subnetwork.

  • A synthetic dataset for 3D lane detection [repo] [data].

Another baseline

This repo also includes an unofficial implementation of '3D-LaneNet' in pytorch for comparison. The method refers to

"3d-lanenet: end-to-end 3d multiple lane detection", N. Garnet, etal., ICCV 2019. [paper]

Requirements

If you have Anaconda installed, you can directly import the provided environment file.

conda env update --file environment.yaml

Those important packages includes:

  • opencv-python 4.1.0.25
  • pytorch 1.4.0
  • torchvision 0.5.0
  • tensorboard 1.15.0
  • tensorboardx 1.7
  • py3-ortools 5.1.4041

Data preparation

The 3D lane detection method is trained and tested on the 3D lane synthetic dataset. Running the demo code on a single image should directly work. However, repeating the training, testing and evaluation requires to prepare the dataset:

If you prefer to build your own data splits using the dataset, please follow the steps described in the 3D lane synthetic dataset repository. All necessary codes are included here already.

Run the Demo

python main_demo_GenLaneNet_ext.py

Specifically, this code predict 3D lane from an image given known camera height and pitch angle. Pretrained models for the segmentation subnetwork and the 3D geometry subnetwork are loaded. Meanwhile, anchor normalization parameters wrt. the training set are also loaded. The demo code will produce lane predication from a single image visualized in the following figure.

The lane results are visualized in three coordinate frames, respectively image plane, virtual top-view, and ego-vehicle coordinate frame. The lane-lines are shown in the top row and the center-lines are shown in the bottom row.

How to train the model

Step 1: Train the segmentation subnetwork

The training of Gen-LaneNet requires to first train the segmentation subnetwork, ERFNet.

  • The training of the ERFNet is based on a pytorch implementation [repo] modified to train the model on the 3D lane synthetic dataset.

  • The trained model should be saved as 'pretrained/erfnet_model_sim3d.tar'. A pre-trained model is already included.

Step 2: Train the 3D-geometry subnetwork

python main_train_GenLaneNet_ext.py
  • Set 'args.dataset_name' to a certain data split to train the model.
  • Set 'args.dataset_dir' to the folder saving the raw dataset.
  • The trained model will be saved in the directory corresponding to certain data split and model name, e.g. 'data_splits/illus_chg/Gen_LaneNet_ext/model*'.
  • The anchor offset std will be recorded for certain data split at the same time, e.g. 'data_splits/illus_chg/geo_anchor_std.json'.

The training progress can be monitored by tensorboard as follows.

cd datas_splits/Gen_LaneNet_ext
./tensorboard  --logdir ./

Batch testing

python main_test_GenLaneNet_ext.py
  • Set 'args.dataset_name' to a certain data split to test the model.
  • Set 'args.dataset_dir' to the folder saving the raw dataset.

The batch testing code not only produces the prediction results, e.g., 'data_splits/illus_chg/Gen_LaneNet_ext/test_pred_file.json', but also perform full-range precision-recall evaluation to produce AP and max F-score.

Other methods

In './experiments', we include the training codes for other variants of Gen-LaneNet models as well as for the baseline method 3D-LaneNet as well as its extended version integrated with the new anchor proposed in Gen-LaneNet. Interested users are welcome to repeat the full set of ablation study reported in the gen-lanenet paper. For example, to train 3D-LaneNet:

cd experiments
python main_train_3DLaneNet.py

Evaluation

Stand-alone evaluation can also be performed.

cd tools
python eval_3D_lane.py

Basically, you need to set 'method_name' and 'data_split' properly to compare the predicted lanes against ground-truth lanes. Evaluation details can refer to the 3D lane synthetic dataset repository or the Gen-LaneNet paper. Overall, the evaluation metrics include:

  • Average Precision (AP)
  • max F-score
  • x-error in close range (0-40 m)
  • x-error in far range (40-100 m)
  • z-error in close range (0-40 m)
  • z-error in far range (40-100 m)

We show the evaluation results comparing two methods:

  • "3d-lanenet: end-to-end 3d multiple lane detection", N. Garnet, etal., ICCV 2019
  • "Gen-lanenet: a generalized and scalable approach for 3D lane detection", Y. Guo, etal., Arxiv, 2020 (GenLaneNet_ext in code)

Comparisons are conducted under three distinguished splits of the dataset. For simplicity, only lane-line results are reported here. The results from the code could be marginally different from that reported in the paper due to different random splits.

  • Standard
Method AP F-Score x error near (m) x error far (m) z error near (m) z error far (m)
3D-LaneNet 89.3 86.4 0.068 0.477 0.015 0.202
Gen-LaneNet 90.1 88.1 0.061 0.496 0.012 0.214
  • Rare Subset
Method AP F-Score x error near (m) x error far (m) z error near (m) z error far (m)
3D-LaneNet 74.6 72.0 0.166 0.855 0.039 0.521
Gen-LaneNet 79.0 78.0 0.139 0.903 0.030 0.539
  • Illumination Change
Method AP F-Score x error near (m) x error far (m) z error near (m) z error far (m)
3D-LaneNet 74.9 72.5 0.115 0.601 0.032 0.230
Gen-LaneNet 87.2 85.3 0.074 0.538 0.015 0.232

Visualization

Visual comparisons to the ground truth can be generated per image when setting 'vis = True' in 'tools/eval_3D_lane.py'. We show two examples for each method under the data split involving illumination change.

  • 3D-LaneNet

  • Gen-LaneNet

Citation

Please cite the paper in your publications if it helps your research:

@article{guo2020gen,
  title={Gen-LaneNet: A Generalized and Scalable Approach for 3D Lane Detection},
  author={Yuliang Guo, Guang Chen, Peitao Zhao, Weide Zhang, Jinghao Miao, Jingao Wang, and Tae Eun Choe},
  booktitle={Computer Vision - {ECCV} 2020 - 16th European Conference},
  year={2020}
}

Copyright and License

The copyright of this work belongs to Baidu Apollo, which is provided under the Apache-2.0 license.

Owner
Yuliang Guo
Researcher in Computer Vision
Yuliang Guo
Fader Networks: Manipulating Images by Sliding Attributes - NIPS 2017

FaderNetworks PyTorch implementation of Fader Networks (NIPS 2017). Fader Networks can generate different realistic versions of images by modifying at

Facebook Research 753 Dec 23, 2022
Based on Yolo's low-power, ultra-lightweight universal target detection algorithm, the parameter is only 250k, and the speed of the smart phone mobile terminal can reach ~300fps+

Based on Yolo's low-power, ultra-lightweight universal target detection algorithm, the parameter is only 250k, and the speed of the smart phone mobile terminal can reach ~300fps+

567 Dec 26, 2022
A simple interface for editing natural photos with generative neural networks.

Neural Photo Editor A simple interface for editing natural photos with generative neural networks. This repository contains code for the paper "Neural

Andy Brock 2.1k Dec 29, 2022
Implementation of H-UCRL Algorithm

Implementation of H-UCRL Algorithm This repository is an implementation of the H-UCRL algorithm introduced in Curi, S., Berkenkamp, F., & Krause, A. (

Sebastian Curi 25 May 20, 2022
Creative Applications of Deep Learning w/ Tensorflow

Creative Applications of Deep Learning w/ Tensorflow This repository contains lecture transcripts and homework assignments as Jupyter Notebooks for th

Parag K Mital 1.5k Dec 30, 2022
Official code of Team Yao at Multi-Modal-Fact-Verification-2022

Official code of Team Yao at Multi-Modal-Fact-Verification-2022 A Multi-Modal Fact Verification dataset released as part of the De-Factify workshop in

Wei-Yao Wang 11 Nov 15, 2022
object recognition with machine learning on Respberry pi

Respberrypi_object-recognition object recognition with machine learning on Respberry pi line.py 建立一支與樹梅派連線的 linebot 使用此 linebot 遠端控制樹梅派拍照 config.ini l

1 Dec 11, 2021
Official implement of Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer

Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer This repository contains the PyTorch code for Evo-ViT. This work proposes a slow-fas

YifanXu 53 Dec 05, 2022
Robust Self-augmentation for NER with Meta-reweighting

Robust Self-augmentation for NER with Meta-reweighting

Lam chi 17 Nov 22, 2022
Evolutionary Scale Modeling (esm): Pretrained language models for proteins

Evolutionary Scale Modeling This repository contains code and pre-trained weights for Transformer protein language models from Facebook AI Research, i

Meta Research 1.6k Jan 09, 2023
Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks

Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks This repository contains a TensorFlow implementation of "

Jingwei Zheng 5 Jan 08, 2023
source code of “Visual Saliency Transformer” (ICCV2021)

Visual Saliency Transformer (VST) source code for our ICCV 2021 paper “Visual Saliency Transformer” by Nian Liu, Ni Zhang, Kaiyuan Wan, Junwei Han, an

89 Dec 21, 2022
A Multi-attribute Controllable Generative Model for Histopathology Image Synthesis

A Multi-attribute Controllable Generative Model for Histopathology Image Synthesis This is the pytorch implementation for our MICCAI 2021 paper. A Mul

Jiarong Ye 7 Apr 04, 2022
Depth-Aware Video Frame Interpolation (CVPR 2019)

DAIN (Depth-Aware Video Frame Interpolation) Project | Paper Wenbo Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang IEEE C

Wenbo Bao 7.7k Dec 31, 2022
Digital Twin Mobility Profiling: A Spatio-Temporal Graph Learning Approach

Digital Twin Mobility Profiling: A Spatio-Temporal Graph Learning Approach This is the implementation of traffic prediction code in DTMP based on PyTo

chenxin 1 Dec 19, 2021
Repo for "Physion: Evaluating Physical Prediction from Vision in Humans and Machines" submission to NeurIPS 2021 (Datasets & Benchmarks track)

Physion: Evaluating Physical Prediction from Vision in Humans and Machines This repo contains code and data to reproduce the results in our paper, Phy

Cognitive Tools Lab 38 Jan 06, 2023
Code for database and frontend of webpage for Neural Fields in Visual Computing and Beyond.

Neural Fields in Visual Computing—Complementary Webpage This is based on the amazing MiniConf project from Hendrik Strobelt and Sasha Rush—thank you!

Brown University Visual Computing Group 29 Nov 30, 2022
Annotated, understandable, and visually interpretable PyTorch implementations of: VAE, BIRVAE, NSGAN, MMGAN, WGAN, WGANGP, LSGAN, DRAGAN, BEGAN, RaGAN, InfoGAN, fGAN, FisherGAN

Overview PyTorch 0.4.1 | Python 3.6.5 Annotated implementations with comparative introductions for minimax, non-saturating, wasserstein, wasserstein g

Shayne O'Brien 471 Dec 16, 2022
Detail-Preserving Transformer for Light Field Image Super-Resolution

DPT Official Pytorch implementation of the paper "Detail-Preserving Transformer for Light Field Image Super-Resolution" accepted by AAAI 2022 . Update

50 Jan 01, 2023