TrackTech: Real-time tracking of subjects and objects on multiple cameras

Overview

TrackTech: Real-time tracking of subjects and objects on multiple cameras

Forwarder Build Interface Build Orchestrator Build Processor Build

Forwarder Docker Pulls Interface Docker Pulls Orchestrator Docker Pulls Processor Docker Pulls

Codecov

DOI

This project is part of the 2021 spring bachelor final project of the Bachelor of Computer Science at Utrecht University. The team that worked on the project consists of eleven students from the Bachelor of Computer Science and Bachelor of Game Technology. This project has been done for educational purposes. All code is open-source, and proper credit is given to respective parties.

GPU support

Updating/Installing drivers

Update the GPU drivers and restart the system for changes to take effect. Optionally, use a different driver listed after running ubuntu-drivers devices

sudo apt install nvidia-driver-460
sudo reboot

Installing the container toolkit

Add the distribution, update the package manager, install NVIDIA for Docker, and restart Docker for changes to take effect. For more information, look at the install guide

distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
   && curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - \
   && curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt update
sudo apt install -y nvidia-docker2
sudo systemctl restart docker

Acquire the GPU ID

According to this read the GPU UUID like GPU-a1b2c3d (just the first part) from

nvidia-smi -a

Add the resource

Add the GPU UUID from the last step to the Docker engine configuration file typically at /etc/docker/daemon.json. Create the file if it does not exist yet.

{
  "runtimes": {
    "nvidia": {
      "path": "/usr/bin/nvidia-container-runtime",
      "runtimeArgs": []
    }
  },
  "default-runtime": "nvidia",
  "node-generic-resources": ["gpu=GPU-a1b2c3d"]
}

Pylint

We use Pylint for python code quality assurance.

Installation

Input following command terminal:

pip install pylint

Run

To run linting on the entire repository, run the following command from the root: pylint CameraProcessor docs Interface ProcessorOrchestrator utility VideoForwarder --rcfile=.pylintrc --reports=n

Explanation

pylint --rcfile=.pylintrc --reports=n

is the Python module to run.

--rcfile is the linting specification used by Pylint.

--reports sets whether the full report should be displayed or not. Our recommendation would be n since this only displays linting errors/warnings and the eventual score.

Constraints

Pylint needs an __init__.py file in the subsystem root to parse all folders to lint. This run must be a subsystem since the root does not contain an __init__.py file.

Ignoring folders from linting

Some folders should be excluded from linting. The exclusion could be for multiple reasons like, the symlinked algorithms in the CameraProcessor folder or the Python virtual environment folder. Add folder name to ignore= in .pylintrc.

Comments
  • FFT: Spc 414 reconnect when stream suddenly stops

    FFT: Spc 414 reconnect when stream suddenly stops

    It works, but there are a lot of but's.

    If the forwarder comes back online and the processor reconnects and starts sending boxes again before the interface reloads, the sync should be fine

    If the forwarder comes back and the interface reloads before the processor starts sending boxes again, sync seems inconsistent. Sometimes it's fine, other times there is a small desync. Desync can be fixed fairly reliably by manually pausing the stream for a few seconds. I think this could be fixed with the 'hack' that makes the video jump a little bit after loading. But this was removed earlier because the jump was considered annoying and not a good fix.

    If the forwarder is not back yet when the interface reloads, there is a chance of ending up with a videojs error and will require a full page reload to fix.

    TL:DR: As long as the processor is sending boxes before the interface reloads sync should remain acceptable

    IMO it's at least better than nothing.

    opened by BrianVanB 2
  • FFT: Remove camera id from the configs.ini

    FFT: Remove camera id from the configs.ini

    Camera id should not be in configurations of cameraprocessor since it is required to be specified inside environment Otherwise it is possible to mistakenly start up camera processor without being guaranteed to have thought about the ID

    opened by GerardvSchie 1
  • Spc 801 pylint enforce class and file name equal

    Spc 801 pylint enforce class and file name equal

    class ITracker requires the file name: i_tracker.py class Tracker requires the file name: tracker.py

    Stricter linting implemented and impacted files are renamed according to the enforced standards

    opened by GerardvSchie 1
  • SPC-728 implement reidentification as a scheduler component

    SPC-728 implement reidentification as a scheduler component

    Extended scheduler to also use globals (objects that do not change during a scheduling iteration (one graph traversal).

    Allow multiple inputs to initial node (initially only 1, but 1 was required).

    Re-id stage and frame buffer (which used output of re-id stage) added to scheduler.

    Only schedules the start node if it is immediately ready, this may or may not be favourable and has the following consequences:

    • Only nodes connected to start node are executed, but only one start node is allowed.
    • If a node is included in plan but only via globals it will not get executed
    opened by tim-van-kemenade 1
  • FIX: SPC 662 fix warning when stream buffers

    FIX: SPC 662 fix warning when stream buffers

    If a stream buffers on first play it would spam the console with a warning saying something is undefined. Fixed by adding more checks if everything is defined before accessing.

    opened by BrianVanB 1
Releases(v1.0.0)
  • v1.0.0(Jun 29, 2021)

    Release v1.0.0

    The following release note will contain a brief overview of each component and its features. Underneath, The currently known bugs can be found.

    Features

    Processor

    The Camera Processor handles the core processing, using detection, tracking, and re-identification algorithms on an image or video feed. It can swap algorithms via new implementations of subclasses of the relevant superclass. Currently implemented are YOLOv5 and YOLOR for detection, SORT for tracking, and TorchReid and FastReid for re-identification.

    Multiple input methods

    The processor processes OpenCV frames. It can process any source that can be turned into a sequence of frames. The supported sources are implemented via a capture interface. The available captures are HLS, video stream, webcam, and an image folder. HLS is how a video feed is received via the internet, which performs extra work to add proper timestamps to the feed.

    Plug and play for main pipeline components

    The main pipeline contains a detection, tracking, and re-identification phase. All these phases are implemented and adhere to the interface belonging to the phase. Implementing another algorithm that conforms to this interface would allow for the algorithm to be loaded in via the configuration. This way, many different algorithms can be defined and swapped when needed.

    Scheduler

    Create a node structure representing a graph, and the scheduler will handle the scheduling of all nodes in each graph iteration. This prevents rewriting things like the pipeline for a more significant change in the form of the pipeline. These graphs are called plans, and thus multiple self-contained plans can be created that can also be swapped on-premise.

    Multiple output methods, deploy opencv tornado

    The processor has three output methods: deploy, opencv, and tornado. Deploy sends information about the processed frame to the orchestrator, which sends it to other processors or the interface. OpenCV displays the processed frames in an OpenCV window. Tornado displays the same OpenCV output but does so in a dedicated webpage. It is discouraged to use the tornado mode for anything other than development since it takes a heavy toll on performance.

    Training of algorithms

    Both the detection and the re-identifications algorithms can be trained with custom datasets. Instructions on how to train these individual components can be found here. The tracking is not based on a neural-network-based implementation and can therefore not be trained.

    Accuracy measurement and metrics

    Several metrics were implemented for determining the accuracy of the detection, the tracking and the re-identification. The detection uses the Mean Average Precision metric. The tracking uses the MOT metric. The re-identification uses the Mean Average Precision and Rank-1 metrics. An extensive explanation of the used accuracy metrics can be found here

    Interface

    A tornado-based webpage interface is used to view the video feeds as well as the detected bounding boxes. It features automated syncing for different camera feeds and their bounding boxes. It has options to select classification types to detect and swap camera focus. The user can click on a bounding box to start tracking an object. The interface features a timeline that keeps track of when and for how long a subject has appeared on each camera for a clear overview.

    Automated bounding box syncing

    When the interface received bounding boxes from the orchestrator and a video stream from the forwarder, it will try to match each box to the frame it belongs to. This is done internally using frame ids. This prevents the user from manually setting the box/video delay to synchronize them.

    Timelines

    Timelines is a page where the history of all tracked objects can be found back. This can be useful to see where an object was during the time it was tracked. When an object is still being tracked, the cutout will be visible next to the object id.

    Forwarder

    Adaptive bitrate

    The forwarder can convert a single incoming stream (like RTMP or RTSP) to multiple bitrate output streams. This way, the stream bitrate can be adapted according to available bandwidth.

    Other

    Security

    OAuth2 is used to make sure only authorized people can access services they should be able to access. Using authentication is optional and can be ignored when developing or testing.

    Docker Images

    Each component contains a Dockerfile used to build images. These images are publicly available on Dockerhub. This allows for easy downloading and deployment.

    Known bugs

    Syncing

    The synchronization of the bounding boxes and the video stream on the interface sometimes mismatch, causing the bounding boxes to have an offset compared to the expected location. Sometimes this can be fixed by pausing the video for a few seconds, but not always.

    Authentication between processor and forwarder

    The OpenCV library to pull in the video from the forwarder does not allow any header to be added to the requests. This means that authentication needs to be disabled for local requests. Luckily most orchestration tools (like docker swarm) allow for a selective port opening to the outside. We allowed unauthenticated forwarder access over port 80 on HTTP (as auth should not be done over an unencrypted connection), which can be used by the processors.

    Processor does not properly handle memory paging on some computers

    This issue only occurred on one computer which had too little memory too handle the processor. The team could not reproduce the bug on other computers that had memory constraints. On this computer, the paging file size keeps increasing until there is no more disk space left, eventually resulting in a processor crash. The processors memory profile does not grow over time thus a system that has enough memory to run for 10 minutes should be able to run for 24 hours or longer. The only memory consumption increasing over time is the feature maps of tracked objects. But these vectors take up little space, and it is generally expected that there are not that many tracked objects.

    Source code(tar.gz)
    Source code(zip)
Binary classification for arrythmia detection with ECG datasets.

HEART DISEASE AI DATATHON 2021 [Eng] / [Kor] #English This is an AI diagnosis modeling contest that uses the heart disease echocardiography and electr

HY_Kim 3 Jul 14, 2022
BigDetection: A Large-scale Benchmark for Improved Object Detector Pre-training

BigDetection: A Large-scale Benchmark for Improved Object Detector Pre-training By Likun Cai, Zhi Zhang, Yi Zhu, Li Zhang, Mu Li, Xiangyang Xue. This

290 Dec 29, 2022
Code for Contrastive-Geometry Networks for Generalized 3D Pose Transfer

Code for Contrastive-Geometry Networks for Generalized 3D Pose Transfer

18 Jun 28, 2022
《LXMERT: Learning Cross-Modality Encoder Representations from Transformers》(EMNLP 2020)

The Most Important Thing. Our code is developed based on: LXMERT: Learning Cross-Modality Encoder Representations from Transformers

53 Dec 16, 2022
GeoMol: Torsional Geometric Generation of Molecular 3D Conformer Ensembles

GeoMol: Torsional Geometric Generation of Molecular 3D Conformer Ensembles This repository contains a method to generate 3D conformer ensembles direct

127 Dec 20, 2022
Official PyTorch implementation of the paper "Self-Supervised Relational Reasoning for Representation Learning", NeurIPS 2020 Spotlight.

Official PyTorch implementation of the paper: "Self-Supervised Relational Reasoning for Representation Learning" (2020), Patacchiola, M., and Storkey,

Massimiliano Patacchiola 135 Jan 03, 2023
TensorFlow code for the neural network presented in the paper: "Structural Language Models of Code" (ICML'2020)

SLM: Structural Language Models of Code This is an official implementation of the model described in: "Structural Language Models of Code" [PDF] To ap

73 Nov 06, 2022
Alphabetical Letter Recognition

DecisionTrees-Image-Classification Alphabetical Letter Recognition In these demo we are using "Decision Trees" Our database is composed by Learning Im

Mohammed Firass 4 Nov 30, 2021
Sharpened cosine similarity torch - A Sharpened Cosine Similarity layer for PyTorch

Sharpened Cosine Similarity A layer implementation for PyTorch Install At your c

Brandon Rohrer 203 Nov 30, 2022
TransPrompt - Towards an Automatic Transferable Prompting Framework for Few-shot Text Classification

TransPrompt This code is implement for our EMNLP 2021's paper 《TransPrompt:Towards an Automatic Transferable Prompting Framework for Few-shot Text Cla

WangJianing 23 Dec 21, 2022
City-Scale Multi-Camera Vehicle Tracking Guided by Crossroad Zones Code

City-Scale Multi-Camera Vehicle Tracking Guided by Crossroad Zones Requirements Python 3.8 or later with all requirements.txt dependencies installed,

88 Dec 12, 2022
A complete end-to-end demonstration in which we collect training data in Unity and use that data to train a deep neural network to predict the pose of a cube. This model is then deployed in a simulated robotic pick-and-place task.

Object Pose Estimation Demo This tutorial will go through the steps necessary to perform pose estimation with a UR3 robotic arm in Unity. You’ll gain

Unity Technologies 187 Dec 24, 2022
Steer OpenAI's Jukebox with Music Taggers

TagBox Steer OpenAI's Jukebox with Music Taggers! The closest thing we have to VQGAN+CLIP for music! Unsupervised Source Separation By Steering Pretra

Ethan Manilow 34 Nov 02, 2022
Code repo for "FASA: Feature Augmentation and Sampling Adaptation for Long-Tailed Instance Segmentation" (ICCV 2021)

FASA: Feature Augmentation and Sampling Adaptation for Long-Tailed Instance Segmentation (ICCV 2021) This repository contains the implementation of th

Yuhang Zang 21 Dec 17, 2022
An implementation of a sequence to sequence neural network using an encoder-decoder

Keras implementation of a sequence to sequence model for time series prediction using an encoder-decoder architecture. I created this post to share a

Luke Tonin 195 Dec 17, 2022
Converting CPT to bert form for use

cpt-encoder 将CPT转成bert形式使用 说明 刚刚刷到又出了一种模型:CPT,看论文显示,在很多中文任务上性能比mac bert还好,就迫不及待想把它用起来。 根据对源码的研究,发现该模型在做nlu建模时主要用的encoder部分,也就是bert,因此我将这部分权重转为bert权重类型

黄辉 1 Oct 14, 2021
Our CIKM21 Paper "Incorporating Query Reformulating Behavior into Web Search Evaluation"

Reformulation-Aware-Metrics Introduction This codebase contains source-code of the Python-based implementation of our CIKM 2021 paper. Chen, Jia, et a

xuanyuan14 5 Mar 05, 2022
Single Image Super-Resolution (SISR) with SRResNet, EDSR and SRGAN

Single Image Super-Resolution (SISR) with SRResNet, EDSR and SRGAN Introduction Image super-resolution (SR) is the process of recovering high-resoluti

8 Apr 15, 2022
Diffusion Probabilistic Models for 3D Point Cloud Generation (CVPR 2021)

Diffusion Probabilistic Models for 3D Point Cloud Generation [Paper] [Code] The official code repository for our CVPR 2021 paper "Diffusion Probabilis

Shitong Luo 323 Jan 05, 2023
Group R-CNN for Point-based Weakly Semi-supervised Object Detection (CVPR2022)

Group R-CNN for Point-based Weakly Semi-supervised Object Detection (CVPR2022) By Shilong Zhang*, Zhuoran Yu*, Liyang Liu*, Xinjiang Wang, Aojun Zhou,

Shilong Zhang 129 Dec 24, 2022