Traffic4D: Single View Reconstruction of Repetitious Activity Using Longitudinal Self-Supervision

Overview

Traffic4D: Single View Reconstruction of Repetitious Activity Using Longitudinal Self-Supervision

Project | PDF | Poster
Fangyu Li, N. Dinesh Reddy, Xudong Chen and Srinivasa G. Narasimhan
Proceedings of IEEE Intelligent Vehicles Symposium (IV'21)
Best Paper Award

Following instructions below, the user will get keypoints, trajectory reconstruction and vehicular activity clustering results like

Set up

The set up process can be skipped if using docker. Please check "Docker" section.

Python

Python version 3.6.9 is used. Python packages are in requirements.txt .

git clone https://github.com/Emrys-Lee/Traffic4D-Release.git
sudo apt-get install python3.6
sudo apt-get install python3-pip
cd Traffic4D-Release
pip3 install -r requirements.txt

C++

Traffic4D uses C++ libraries ceres and pybind for efficient optimization. pybind needs clang compiler, so Traffic4D uses clang compiler.

Install clang compiler

sudo apt-get install clang++-6.0

Install prerequisites for ceres

# CMake
sudo apt-get install cmake
# google-glog + gflags
sudo apt-get install libgoogle-glog-dev libgflags-dev
# BLAS & LAPACK
sudo apt-get install libatlas-base-dev
# Eigen3
sudo apt-get install libeigen3-dev
# SuiteSparse and CXSparse (optional)
sudo apt-get install libsuitesparse-dev

Download and install ceres

wget https://github.com/ceres-solver/ceres-solver/archive/1.12.0.zip
unzip 1.12.0.zip
cd ceres-solver-1.12.0/
mkdir build
cd build
cmake ..
make
sudo make install

Download and install pybind

git clone https://github.com/pybind/pybind11
cd pybind11
cmake .
make
sudo make install

Build Traffic4D optimization library

cd Traffic4D-Release/src/ceres
make

ceres_reconstruct.so and ceres_spline.so are generated under path Traffic4D-Release/src/ceres/.

Dataset

Download dataset and pre-generated results from here, and put it under Traffic4D-Release/.

cd Traffic4D-Release
mv Data-Traffic4D.zip ./
unzip Data-Traffic4D.zip

The directory should be like

Traffic4D-Release/
    Data-Traffic4D/
    └───fifth_morewood/
        └───fifth_morewood_init.vd
        └───top_view.png
        └───images/
                00001.jpg
                00002.jpg
                ...
                06288.jpg
    └───arterial_kennedy/
        └───arterial_kennedy_init.vd
        └───top_view.png
        └───images/
                <put AI City Challenge frames here>
        ...

The input and output paths can be modified in config/*.yml.

Explanation

1. Input videos

Sample videos in Traffic4D are provided. Note arterial_kennedy and dodge_century are from Nvidia AI City Challenge City-Scale Multi-Camera Vehicle Tracking Challenge Track. Please request the access to the dataset here. Once get the data, run

ffmpeg -i <mtmc-dir>/train/S01/c001/vdo.avi Traffic4D-Release/Data-Traffic4D/arterial_kennedy/images/%05d.jpg
ffmpeg -i <mtmc-dir>/test/S02/c007/vdo.avi Traffic4D-Release/Data-Traffic4D/dodge_century/images/%05d.jpg

to extract frames into images/.

2. Pre-Generated 2D results

Detected 2D bounding boxes, keypoints and tracking IDs are stored in *_init.vd. Check Occlusionnet implementation for detecting keypoints; V-IOU for multi-object tracking.

3. Output folder

Folder Traffic4D-Release/Result/ will be created by default.

Experiments

Run python exp/traffic4d.py config/<intersection_name>.yml <action>. Here YML configuration files for multiple intersections are provided under config/ folder. <action> shoulbe be reconstruction or clustering to perform longitudinal reconstruction and activity clustering sequentially. For example, below runs Fifth and Morewood intersection.

cd Traffic4D-Release
python3 exp/traffic4d.py config/fifth_morewood.yml reconstruction
python3 exp/traffic4d.py config/fifth_morewood.yml clustering

Results

Find these results in the output folder:

  1. 2D keypoints: If 3D reconstruction is done, 2D reprojected keypoints will be plotted in Traffic4D-Release/Result/<intersection_name>_keypoints/.
  2. 3D reconstructed trajectories and clusters: The clustered 3D trajectories are plotted on the top view map as Traffic4D-Release/Result/<intersection_name>_top_view.jpg.

Docker

We provide docker image with dependencies already set up. The steps in "Set up" can be skipped if you use docker image. You still need to clone the repo and download the dataset and put it in under Traffic4D-Release/.

git clone https://github.com/Emrys-Lee/Traffic4D-Release.git

Pull Traffic4D docker image.

docker pull emrysli/traffic4d-release:latest

Then create a container and map the git repo into docker container to access the dataset. For example, if the cloned repo locates at host directory /home/xxx/Traffic4D-Release, <path_to_repo> should be /home/xxx. If <path_in_container> is /home/yyy, then /home/xxx/Traffic4D-Release will be mapped as /home/yyy/Traffic4D-Release inside the container.

docker run -it -v <path_to_repo>/Traffic4D-Release:<path_in_container>/Traffic4D-Release emrysli/traffic4d-release:latest /bin/bash

Inside container compile Traffic4D again.

# inside container
cd <path_in_container>/Traffic4D-Release/src/ceres
make

Run experiments.

cd <path_in_container>/Traffic4D-Release
python3 exp/traffic4d.py config/fifth_morewood.yml reconstruction
python3 exp/traffic4d.py config/fifth_morewood.yml clustering

Trouble Shooting

  1. tkinter module is missing
File "/usr/local/lib/python3.6/dist-packages/matplotlib/backends/_backend_tk.py", line 5, in <module>
    import tkinter as Tk
ModuleNotFoundError: No module named 'tkinter'

Solution: install tkinter.

sudo apt-get install python3-tk
  1. opencv import error such as
File "/usr/local/lib/python3.6/dist-packages/cv2/__init__.py", line 3, in <module>
    from .cv2 import *
ImportError: libSM.so.6: cannot open shared object file: No such file or directory

Solution: install the missing libraries.

sudo apt-get install libsm6 libxrender1 libfontconfig1 libxext6

Citation

Traffic4D

@conference{Li-2021-127410,
author = {Fangyu Li and N. Dinesh Reddy and Xudong Chen and Srinivasa G. Narasimhan},
title = {Traffic4D: Single View Reconstruction of Repetitious Activity Using Longitudinal Self-Supervision},
booktitle = {Proceedings of IEEE Intelligent Vehicles Symposium (IV '21)},
year = {2021},
month = {July},
publisher = {IEEE},
keywords = {Self-Supervision, vehicle Detection, 4D Reconstruction, 3D reconstuction, Pose Estimation.},
}

Occlusion-Net

@inproceedings{onet_cvpr19,
author = {Reddy, N. Dinesh and Vo, Minh and Narasimhan, Srinivasa G.},
title = {Occlusion-Net: 2D/3D Occluded Keypoint Localization Using Graph Networks},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
pages = {7326--7335},
year = {2019}
}
This is the implementation of the paper LiST: Lite Self-training Makes Efficient Few-shot Learners.

LiST (Lite Self-Training) This is the implementation of the paper LiST: Lite Self-training Makes Efficient Few-shot Learners. LiST is short for Lite S

Microsoft 28 Dec 07, 2022
As-ViT: Auto-scaling Vision Transformers without Training

As-ViT: Auto-scaling Vision Transformers without Training [PDF] Wuyang Chen, Wei Huang, Xianzhi Du, Xiaodan Song, Zhangyang Wang, Denny Zhou In ICLR 2

VITA 68 Sep 05, 2022
Annotate with anyone, anywhere.

h h is the web app that serves most of the https://hypothes.is/ website, including the web annotations API at https://hypothes.is/api/. The Hypothesis

Hypothesis 2.6k Jan 08, 2023
Toolbox of models, callbacks, and datasets for AI/ML researchers.

Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch Website • Installation • Main

Pytorch Lightning 1.4k Dec 30, 2022
RM Operation can equivalently convert ResNet to VGG, which is better for pruning; and can help RepVGG perform better when the depth is large.

RM Operation can equivalently convert ResNet to VGG, which is better for pruning; and can help RepVGG perform better when the depth is large.

184 Jan 04, 2023
ConE: Cone Embeddings for Multi-Hop Reasoning over Knowledge Graphs

ConE: Cone Embeddings for Multi-Hop Reasoning over Knowledge Graphs This is the code of paper ConE: Cone Embeddings for Multi-Hop Reasoning over Knowl

MIRA Lab 33 Dec 07, 2022
Place holder for HOPE: a human-centric and task-oriented MT evaluation framework using professional post-editing

HOPE: A Task-Oriented and Human-Centric Evaluation Framework Using Professional Post-Editing Towards More Effective MT Evaluation Place holder for dat

Lifeng Han 1 Apr 25, 2022
Joint detection and tracking model named DEFT, or ``Detection Embeddings for Tracking.

DEFT: Detection Embeddings for Tracking DEFT: Detection Embeddings for Tracking, Mohamed Chaabane, Peter Zhang, J. Ross Beveridge, Stephen O'Hara

Mohamed Chaabane 253 Dec 18, 2022
Probabilistic Entity Representation Model for Reasoning over Knowledge Graphs

Implementation for the paper: Probabilistic Entity Representation Model for Reasoning over Knowledge Graphs, Nurendra Choudhary, Nikhil Rao, Sumeet Ka

Nurendra Choudhary 8 Nov 15, 2022
Non-Attentive-Tacotron - This is Pytorch Implementation of Google's Non-attentive Tacotron.

Non-attentive Tacotron - PyTorch Implementation This is Pytorch Implementation of Google's Non-attentive Tacotron, text-to-speech system. There is som

Jounghee Kim 46 Dec 19, 2022
The VeriNet toolkit for verification of neural networks

VeriNet The VeriNet toolkit is a state-of-the-art sound and complete symbolic interval propagation based toolkit for verification of neural networks.

9 Dec 21, 2022
we propose EfficientDerain for high-efficiency single-image deraining

EfficientDerain we propose EfficientDerain for high-efficiency single-image deraining Requirements python 3.6 pytorch 1.6.0 opencv-python 4.4.0.44 sci

Qing Guo 126 Dec 07, 2022
MvtecAD unsupervised Anomaly Detection

MvtecAD unsupervised Anomaly Detection This respository is the unofficial implementations of DFR: Deep Feature Reconstruction for Unsupervised Anomaly

0 Feb 25, 2022
Keras code and weights files for popular deep learning models.

Trained image classification models for Keras THIS REPOSITORY IS DEPRECATED. USE THE MODULE keras.applications INSTEAD. Pull requests will not be revi

François Chollet 7.2k Dec 29, 2022
Pytorch implementation for the Temporal and Object Quantification Networks (TOQ-Nets).

TOQ-Nets-PyTorch-Release Pytorch implementation for the Temporal and Object Quantification Networks (TOQ-Nets). Temporal and Object Quantification Net

Zhezheng Luo 9 Jun 30, 2022
FAST Aiming at the problems of cumbersome steps and slow download speed of GNSS data

FAST Aiming at the problems of cumbersome steps and slow download speed of GNSS data, a relatively complete set of integrated multi-source data download terminal software fast is developed. The softw

ChangChuntao 23 Dec 31, 2022
ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models (ICCV 2021 Oral)

ILVR + ADM This is the implementation of ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models (ICCV 2021 Oral). This repository is h

Jooyoung Choi 225 Dec 28, 2022
LaneAF: Robust Multi-Lane Detection with Affinity Fields

LaneAF: Robust Multi-Lane Detection with Affinity Fields This repository contains Pytorch code for training and testing LaneAF lane detection models i

155 Dec 17, 2022
Implementations of orthogonal and semi-orthogonal convolutions in the Fourier domain with applications to adversarial robustness

Orthogonalizing Convolutional Layers with the Cayley Transform This repository contains implementations and source code to reproduce experiments for t

CMU Locus Lab 36 Dec 30, 2022
PyTorch Implementation of Region Similarity Representation Learning (ReSim)

ReSim This repository provides the PyTorch implementation of Region Similarity Representation Learning (ReSim) described in this paper: @Article{xiao2

Tete Xiao 74 Jan 03, 2023