[arXiv] What-If Motion Prediction for Autonomous Driving ❓🚗💨

Last update: Dec 29, 2022

Overview

WIMP - What If Motion Predictor

Reference PyTorch Implementation for What If Motion Prediction [PDF] [Dynamic Visualizations]

Setup

Requirements

The WIMP reference implementation and setup procedure has been tested to work with Ubuntu 16.04+ and has the following requirements:

python >= 3.7
pytorch >= 1.5.0

Installing Dependencies

Install remaining required Python dependencies using pip.
```
pip install -r requirements.txt
```
Install the Argoverse API module into the local Python environment by following steps 1, 2, and 4 in the README.

Argoverse Data

In order to set up the Argoverse dataset for training and evaluation, follow the steps below:

Download the the Argoverse Motion Forecasting v1.1 dataset and extract the compressed data subsets such that the raw CSV files are stored in the following directory structure:

├── WIMP
│   ├── src
│   ├── scripts
│   ├── data
│   │   ├── argoverse_raw
│   │   │   ├── train
│   │   │   │   ├── *.csv
│   │   │   ├── val
│   │   │   │   ├── *.csv
│   │   │   ├── test
│   │   │   │   ├── *.csv

Pre-process the raw Argoverse data into a WIMP-compatible format by running the following script. It should be noted that the Argoverse dataset is quite large and this script may take a few hours to run on a multi-threaded machine.

python scripts/run_preprocess.py --dataroot ./data/argoverse_raw/ \
--mode val --save-dir ./data/argoverse_processed --social-features \
--map-features --xy-features --normalize --extra-map-features \
--compute-all --generate-candidate-centerlines 6

Usage

For a detailed description of all possible configuration arguments, please run scripts with the -h flag.

Training

To train WIMP from scratch using a configuration similar to that reported in the paper, run a variant of the following command:

python src/main.py --mode train --dataroot ./data/argoverse_processed --IFC \
--lr 0.0001 --weight-decay 0.0 --non-linearity relu  --use-centerline-features \
--segment-CL-Encoder-Prob --num-mixtures 6 --output-conv --output-prediction \
--gradient-clipping --hidden-key-generator --k-value-threshold 10 \
--scheduler-step-size 60 90 120 150 180  --distributed-backend ddp \
--experiment-name example --gpus 4 --batch-size 25

Citing

If you've found this code to be useful, please consider citing our paper!

@article{khandelwal2020if,
  title={What-If Motion Prediction for Autonomous Driving},
  author={Khandelwal, Siddhesh and Qi, William and Singh, Jagjeet and Hartnett, Andrew and Ramanan, Deva},
  journal={arXiv preprint arXiv:2008.10587},
  year={2020}
}

Questions

This repo is maintained by William Qi and Siddhesh Khandelwal - please feel free to reach out or open an issue if you have additional questions/concerns.

We plan to clean up the codebase and add some additional utilities (possibly NuScenes data loaders and inference/visualization tools) in the near future, but don't expect to make significant breaking changes.

Comments

Pandas Error runpreprocess.py

Hello! First of all, thank you for making your code available for the readers of your great paper. I am having an issue while running run_preprocess.py. I think while reading the csv something goes wrong since my error is a pandas error. When I try to run the script, it gives me: KeyError: 'CITY_NAME' When I go to the script and give "MIA" as the CITY_NAME, just to see what happens, I receive a similar error: KeyError: 'OBJECT_TYPE' I checked the paths for the data. It seems fine. What could be the reason? Thank you!

opened by ahmetgurhan 0

Loss dimensions

Hi, thank you so much for your fantastic work.

Which is the order, and the dimensions, in this function?

def l1_ewta_loss(prediction, target, k=6, eps=1e-7, mr=2.0):
    num_mixtures = prediction.shape[1]

    target = target.unsqueeze(1).expand(-1, num_mixtures, -1, -1)
    l1_loss = nn.functional.l1_loss(prediction, target, reduction='none').sum(dim=[2, 3])

    # Get loss from top-k mixtures for each timestep
    mixture_loss_sorted, mixture_ranks = torch.sort(l1_loss, descending=False)
    mixture_loss_topk = mixture_loss_sorted.narrow(1, 0, k)

    # Aggregate loss across timesteps and batch
    loss = mixture_loss_topk.sum()
    loss = loss / target.size(0)
    loss = loss / target.size(2)
    loss = loss / k
    return loss

I am not able to obtain good results compared to NLL. I have as inputs:

predictions: batch_size x num_modes x pred_len x data_dim (e.g. 1024 x 6 x 30 x 2) gt: batch_size x pred_len x data_dim (e.g. 1024 x 30 x 2)

Is this correct?

opened by Cram3r95 0

Reproducing the Map-Free and only Social-Context Results form the Ablation Study

Hey there,

I want to reproduce the results of your ablation study, where you only used Social-Context with EWTA-Loss.

However, I habe problems training the model only with social context. What are the correct flags I need to set for preprocessing (run_preprocess.py) and for training (main.py)?

Looking forward hearing from you soon!

Best regards

SchDevel

opened by SchDevel 2
Can I get your inference/visualization code?

Hi, first of all, thanks for your awesome work and sharing that to us.

I tried to make inference/visualization code by myself, unfortunately, there were some problems.

Maybe library's mismatching, my insufficient coding skills, or something else.

So, can i get your inference/visualization code or even skeleton base code?

opened by raspbe34 3
What is the method for incomplete trajectories?
Hi, thanks for sharing your great work~ I am wondering how you deal with the incomplete trajectories problem (agents have less then 2 seconds of history).

I notice that for the neighboring agent wrt focal agent, you discard all the agents (code) if their trajectories are not complete

how would you deal with those incomplete trajectories for the focal agent? Did you use interpolation or some techniques?

Thanks!
opened by XHwind 0

Releases(1.0)

1.0(Oct 8, 2020)

Initial code release for WIMP.
Source code(tar.gz)
Source code(zip)

Owner

William Qi

Prediction @argoai

GitHub Repository

Header-only library for using Keras models in C++.

frugally-deep Use Keras models in C++ with ease Table of contents Introduction Usage Performance Requirements and Installation FAQ Introduction Would

927 Jan 05, 2023

Illuminated3D This project participates in the Nasa Space Apps Challenge 2021.

1 Oct 09, 2021

An educational resource to help anyone learn deep reinforcement learning.

Status: Maintenance (expect bug fixes and minor updates) Welcome to Spinning Up in Deep RL! This is an educational resource produced by OpenAI that ma

7.6k Jan 09, 2023

Source code for Transformer-based Multi-task Learning for Disaster Tweet Categorisation (UCD's participation in TREC-IS 2020A, 2020B and 2021A).

Source code for "UCD participation in TREC-IS 2020A, 2020B and 2021A". *** update at: 2021/05/25 This repo so far relates to the following work: Trans

4 Oct 19, 2021

An ML & Correlation platform for transforming disparate data points of interest into usable intelligence.

SSIDprobeCollector An ML & Correlation platform for transforming disparate data points of interest into usable intelligence. At a High level the platf

1 Jan 30, 2022

Monocular Depth Estimation Using Laplacian Pyramid-Based Depth Residuals

LapDepth-release This repository is a Pytorch implementation of the paper "Monocular Depth Estimation Using Laplacian Pyramid-Based Depth Residuals" M

205 Dec 30, 2022

Fast sparse deep learning on CPUs

SPARSEDNN **If you want to use this repo, please send me an email: [email pro

44 Nov 30, 2022

TCube generates rich and fluent narratives that describes the characteristics, trends, and anomalies of any time-series data (domain-agnostic) using the transfer learning capabilities of PLMs.

TCube: Domain-Agnostic Neural Time series Narration This repository contains the code for the paper: "TCube: Domain-Agnostic Neural Time series Narrat

7 Oct 31, 2021

[arXiv] What-If Motion Prediction for Autonomous Driving ❓🚗💨

Related tags

Overview

WIMP - What If Motion Predictor

Setup

Requirements

Installing Dependencies

Argoverse Data

Usage

Training

Citing

Questions

Comments

Pandas Error runpreprocess.py

Loss dimensions

Reproducing the Map-Free and only Social-Context Results form the Ablation Study

Can I get your inference/visualization code?

What is the method for incomplete trajectories?

Releases(1.0)

1.0(Oct 8, 2020)

Owner

William Qi

Header-only library for using Keras models in C++.

Illuminated3D This project participates in the Nasa Space Apps Challenge 2021.

An educational resource to help anyone learn deep reinforcement learning.

Source code for Transformer-based Multi-task Learning for Disaster Tweet Categorisation (UCD's participation in TREC-IS 2020A, 2020B and 2021A).

An ML & Correlation platform for transforming disparate data points of interest into usable intelligence.

Monocular Depth Estimation Using Laplacian Pyramid-Based Depth Residuals

Fast sparse deep learning on CPUs

TCube generates rich and fluent narratives that describes the characteristics, trends, and anomalies of any time-series data (domain-agnostic) using the transfer learning capabilities of PLMs.

Official PyTorch implementation of Data-free Knowledge Distillation for Object Detection, WACV 2021.

Code to reproduce experiments in the paper "Explainability Requires Interactivity".

Data & Code for ACCENTOR Adding Chit-Chat to Enhance Task-Oriented Dialogues

Another pytorch implementation of FCN (Fully Convolutional Networks)

An original implementation of "Noisy Channel Language Model Prompting for Few-Shot Text Classification"

Title: Heart-Failure-Classification

This project aims at building a real-time wide band channel sounder using USRPs

Fbone (Flask bone) is a Flask (Python microframework) starter/template/bootstrap/boilerplate application.

GrailQA: Strongly Generalizable Question Answering

Tech Resources for Academic Communities

A repo that contains all the mesh keys needed for mesh backend, along with a code example of how to use them in python

Python implementation of MULTIseq barcode alignment using fuzzy string matching and GMM barcode assignment