Official PyTorch implementation of Joint Object Detection and Multi-Object Tracking with Graph Neural Networks

Related tags

Deep LearningGSDT
Overview

GSDT

Joint Object Detection and Multi-Object Tracking with Graph Neural Networks

This is the official PyTorch implementation of our paper: "Joint Object Detection and Multi-Object Tracking with Graph Neural Networks". Our project website and video demos are here. If you find our work useful, we'd appreciate you citing our paper as follows:

@article{Wang2020_GSDT, 
author = {Wang, Yongxin and Kitani, Kris and Weng, Xinshuo}, 
journal = {arXiv:2006.13164}, 
title = {{Joint Object Detection and Multi-Object Tracking with Graph Neural Networks}}, 
year = {2020} 
}

Introduction

Object detection and data association are critical components in multi-object tracking (MOT) systems. Despite the fact that the two components are dependent on each other, prior work often designs detection and data association modules separately which are trained with different objectives. As a result, we cannot back-propagate the gradients and optimize the entire MOT system, which leads to sub-optimal performance. To address this issue, recent work simultaneously optimizes detection and data association modules under a joint MOT framework, which has shown improved performance in both modules. In this work, we propose a new instance of joint MOT approach based on Graph Neural Networks (GNNs). The key idea is that GNNs can model relations between variable-sized objects in both the spatial and temporal domains, which is essential for learning discriminative features for detection and data association. Through extensive experiments on the MOT15/16/17/20 datasets, we demonstrate the effectiveness of our GNN-based joint MOT approach and show the state-of-the-art performance for both detection and MOT tasks.

Usage

Dependencies

We recommend using anaconda for managing dependency and environments. You may follow the commands below to setup your environment.

conda create -n dev python=3.6
conda activate dev
pip install -r requirements.txt

We use the PyTorch Geometric package for the implementation of our Graph Neural Network based architecture.

bash install_pyg.sh   # we used CUDA_version=cu101 

Build Deformable Convolutional Networks V2 (DCNv2)

cd ./src/lib/models/networks/DCNv2
bash make.sh

To automatically generate output tracking as videos, please install ffmpeg

conda install ffmpeg=4.2.2

Data preperation

We follow the same dataset setup as in JDE. Please refer to their DATA ZOO for data download and preperation.

To prepare 2DMOT15 and MOT20 data, you can directly download from the MOT Challenge website, and format each directory as follows:

MOT15
   |——————images
   |        └——————train
   |        └——————test
   └——————labels_with_ids
            └——————train(empty)
MOT20
   |——————images
   |        └——————train
   |        └——————test
   └——————labels_with_ids
            └——————train(empty)

Then change the seq_root and label_root in src/gen_labels_15.py and src/gen_labels_20.py accordingly, and run:

cd src
python gen_labels_15.py
python gen_labels_20.py

This will generate the desired label format of 2DMOT15 and MOT20. The seqinfo.ini files are required for 2DMOT15 and can be found here [Google], [Baidu],code:8o0w.

Inference

Download and save the pretrained weights for each dataset by following the links below:

Dataset Model
2DMOT15 model_mot15.pth
MOT17 model_mot17.pth
MOT20 model_mot20.pth

Run one of the following command to reproduce our paper's tracking performance on the MOT Challenge.

cd ./experiments
track_gnn_mot_AGNNConv_RoIAlign_mot15.sh 
track_gnn_mot_AGNNConv_RoIAlign_mot17.sh 
track_gnn_mot_AGNNConv_RoIAlign_mot20.sh 

To clarify, currently we directly used the MOT17 results as MOT16 results for submission. That is, our MOT16 and MOT17 results and models are identical.

Training

We are currently in the process of cleaning the training code. We'll release as soon as we can. Stay tuned!

Performance on MOT Challenge

You can refer to MOTChallenge website for performance of our method. For your convenience, we summarize results below:

Dataset MOTA IDF1 MT ML IDS
2DMOT15 60.7 64.6 47.0% 10.5% 477
MOT16 66.7 69.2 38.6% 19.0% 959
MOT17 66.2 68.7 40.8% 18.3% 3318
MOT20 67.1 67.5 53.1% 13.2% 3133

Acknowledgement

A large part of the code is borrowed from FairMOT. We appreciate their great work!

Owner
Richard Wang
Richard Wang
Identifying Stroke Indicators Using Rough Sets

Identifying Stroke Indicators Using Rough Sets With the spirit of reproducible research, this repository contains all the codes required to produce th

Muhammad Salman Pathan 0 Jun 09, 2022
image scene graph generation benchmark

Scene Graph Benchmark in PyTorch 1.7 This project is based on maskrcnn-benchmark Highlights Upgrad to pytorch 1.7 Multi-GPU training and inference Bat

Microsoft 303 Dec 27, 2022
A colab notebook for training Stylegan2-ada on colab, transfer learning onto your own dataset.

Stylegan2-Ada-Google-Colab-Starter-Notebook A no thrills colab notebook for training Stylegan2-ada on colab. transfer learning onto your own dataset h

Harnick Khera 66 Dec 16, 2022
Multi-Target Adversarial Frameworks for Domain Adaptation in Semantic Segmentation

Multi-Target Adversarial Frameworks for Domain Adaptation in Semantic Segmentation Paper Multi-Target Adversarial Frameworks for Domain Adaptation in

Valeo.ai 20 Jun 21, 2022
Jittor implementation of PCT:Point Cloud Transformer

PCT: Point Cloud Transformer This is a Jittor implementation of PCT: Point Cloud Transformer.

MenghaoGuo 547 Jan 03, 2023
Official implementation for paper Knowledge Bridging for Empathetic Dialogue Generation (AAAI 2021).

Knowledge Bridging for Empathetic Dialogue Generation This is the official implementation for paper Knowledge Bridging for Empathetic Dialogue Generat

Qintong Li 50 Dec 20, 2022
RIM: Reliable Influence-based Active Learning on Graphs.

RIM: Reliable Influence-based Active Learning on Graphs. This repository is the official implementation of RIM. Requirements To install requirements:

Wentao Zhang 4 Aug 29, 2022
PyTorch implementation of image classification models for CIFAR-10/CIFAR-100/MNIST/FashionMNIST/Kuzushiji-MNIST/ImageNet

PyTorch Image Classification Following papers are implemented using PyTorch. ResNet (1512.03385) ResNet-preact (1603.05027) WRN (1605.07146) DenseNet

1.2k Jan 04, 2023
The deployment framework aims to provide a simple, lightweight, fast integrated, pipelined deployment framework that ensures reliability, high concurrency and scalability of services.

savior是一个能够进行快速集成算法模块并支持高性能部署的轻量开发框架。能够帮助将团队进行快速想法验证(PoC),避免重复的去github上找模型然后复现模型;能够帮助团队将功能进行流程拆解,很方便的提高分布式执行效率;能够有效减少代码冗余,减少不必要负担。

Tao Luo 125 Dec 22, 2022
A list of all papers and resoureces on Semantic Segmentation

Semantic-Segmentation A list of all papers and resoureces on Semantic Segmentation. Dataset importance SemanticSegmentation_DL Some implementation of

Alan Tang 1.1k Dec 12, 2022
DeepLab resnet v2 model in pytorch

pytorch-deeplab-resnet DeepLab resnet v2 model implementation in pytorch. The architecture of deepLab-ResNet has been replicated exactly as it is from

Isht Dwivedi 601 Dec 22, 2022
This is code to fit per-pixel environment map with spherical Gaussian lobes, using LBFGS optimization

Spherical Gaussian Optimization This is code to fit per-pixel environment map with spherical Gaussian lobes, using LBFGS optimization. This code has b

41 Dec 14, 2022
Code for our ICCV 2021 Paper "OadTR: Online Action Detection with Transformers".

Code for our ICCV 2021 Paper "OadTR: Online Action Detection with Transformers".

66 Dec 15, 2022
EsViT: Efficient self-supervised Vision Transformers

Efficient Self-Supervised Vision Transformers (EsViT) PyTorch implementation for EsViT, built with two techniques: A multi-stage Transformer architect

Microsoft 352 Dec 25, 2022
Implementation of PyTorch-based multi-task pre-trained models

mtdp Library containing implementation related to the research paper "Multi-task pre-training of deep neural networks for digital pathology" (Mormont

Romain Mormont 27 Oct 14, 2022
Discord-Protect is a simple discord bot allowing you to have some security on your discord server by ordering a captcha to the user who joins your server.

Discord-Protect Discord-Protect is a simple discord bot allowing you to have some security on your discord server by ordering a captcha to the user wh

Tir Omar 2 Oct 28, 2021
A repo to show how to use custom dataset to train s2anet, and change backbone to resnext101

A repo to show how to use custom dataset to train s2anet, and change backbone to resnext101

jedibobo 3 Dec 28, 2022
OMNIVORE is a single vision model for many different visual modalities

Omnivore: A Single Model for Many Visual Modalities [paper][website] OMNIVORE is a single vision model for many different visual modalities. It learns

Meta Research 451 Dec 27, 2022
《Deep Single Portrait Image Relighting》(ICCV 2019)

Ratio Image Based Rendering for Deep Single-Image Portrait Relighting [Project Page] This is part of the Deep Portrait Relighting project. If you find

62 Dec 21, 2022
For IBM Quantum Challenge 2021 (May 20 - 26)

IBM Quantum Challenge 2021 Introduction Commemorating the 40-year anniversary of the Physics of Computation conference, and 5-year anniversary of IBM

Qiskit Community 140 Jan 01, 2023