TorchGRL is the source code for our paper Graph Convolution-Based Deep Reinforcement Learning for Multi-Agent Decision-Making in Mixed Traffic Environments for IV 2022.

Related tags

Deep LearningTorchGRL
Overview

TorchGRL

TorchGRL is the source code for our paper Graph Convolution-Based Deep Reinforcement Learning for Multi-Agent Decision-Making in Mixed Traffic Environments for IV 2022.TorchGRL is a modular simulation framework that integrates different GRL algorithms and SUMO simulation platform to realize the simulation of multi-agents decision-making algorithms in mixed traffic environment. You can adjust the test scenarios and the implemented GRL algorithm according to your needs.


Preparation

Before starting to carry out some relevant works on our framework, some preparations are required to be done.

Hardware

Our framework is developed based on a laptop, and the specific configuration is as follows:

  • Operating system: Ubuntu 20.04
  • RAM: 32 GB
  • CPU: Intel (R) Core (TM) i9-10980HK CPU @ 2.40GHz
  • GPU: RTX 2070

It should be noted that our program must be reproduced under the Ubuntu 20.04 operating system, and we strongly recommend using GPU for training.

Development Environment

Before compiling the code of our framework, you need to install the following development environment:

  • Ubuntu 20.04 with latest GPU driver
  • Pycharm
  • Anaconda
  • CUDA 11.1
  • cudnn-11.1, 8.0.5.39

Installation

Please download our GRL framework repository first:

git clone https://github.com/Jacklinkk/TorchGRL.git

Then enter the root directory of TorchGRL:

cd TorchGRL

and please be sure to run the below commands from /path/to/TorchGRL.

Installation of FLOW

The FLOW library will be firstly installed.

Firstly, enter the flow directory:

cd flow

Then, create a conda environment from flow library:

conda env create -f environment.yml

Activate conda environment:

conda activate TorchGCQ

Install flow from source code:

python setup.py develop

Installation of SUMO

SUMO simulation platform will be installed. Please make sure to run the below commands in the "TorchGRL" virtual environment.

Install via pip:

pip install eclipse-sumo

Setting in Pycharm:

In order to adopt SUMO correctly, you need to define the environment variable of SUMO_HOME in Pycharm. The specific directory is:

/home/…/.conda/envs/TorchGCQ/lib/python3.7/site-packages/sumo

Setting in Ubuntu:

At first, run:

gedit ~/.bashrc

then copy the path name of SUMO_HOME to “~/.bashrc”:

export SUMO_HOME=“/home/…/.conda/envs/TorchGCQ/lib/python3.7/site-packages/sumo”

Finally, run:

source ~/.bashrc

Installation of Pytorch and related libraries

Please make sure to run the below commands in the "TorchGRL" virtual environment.

Installation of Pytorch:

We use Pytorch version 1.9.0 for development under a specific version of CUDA and cudnn.

pip3 install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html

Installation of pytorch geometric:

Pytorch geometric is a Graph Neural Network (GNN) library upon Pytorch

pip install torch-scatter torch-sparse torch-cluster torch-spline-conv torch-geometric -f https://data.pyg.org/whl/torch-1.9.0+cu111.html

Installation of pfrl library

Please make sure to run the below commands in the "TorchGRL" virtual environment.

pfrl is a deep reinforcement learning library that implements various algorithms in Python using PyTorch.

Firstly, enter the pfrl directory:

cd pfrl

Then install from source code:

python setup.py develop

Instruction

flow folder

The flow folder is the root directory of the library after the FLOW library is installed through source code, including interface-related programs between DRL algorithms and SUMO platform.

Flow_Test folder

The Flow_Test folder includes the related programs of the test environment configuration; specifically, T_01.py is the core python program. If the program runs successfully, the environment configuration is successful.

pfrl folder

The pfrl folder is the root directory of the library after the deep reinforcement learning pfrl library is installed through source code, including all DRL related programs. The source program can be modified as needed.

GRLNet folder

The GRLNet folder contains the GRL neural network built in the Pytorch environment. You can modify the source code as needed or add your own neural network.

  • Pytorch_GRL.py constructs the fundamental neural network of GRL algorithms
  • Pytorch_GRL_Dueling.py constructs the dueling network of GRL algorithms

GRL_utils folder

The GRL_utils folder contains basic functions such as model training and testing, data storage, and curve drawing.

  • Train_and_Test.py contains the training and testing functions for the GRL model.
  • Data_Plot_Train.py is the function to plot the training data curve.
  • Data_Process_Test.py is the function to process the test data.
  • Fig folder stores the training data curve.
  • Logging_Training folder stores the training data generated by different GRL algorithms.
  • Logging_Test folder stores the testing data generated by different GRL algorithms.

GRL_Simulation folder

The GRL_Simulation folder is the core of our framework, which contains the core simulation program and some related functional programs.

  • main.py is the main program, containing the definition of FLOW parameters, as well as the controlling (start and end) of the simulation.
  • controller.py is the definition of vehicle control model based on FLOW library.
  • environment.py is the core program to build and initialize the simulation environment of SUMO.
  • network.py defines the road network.
  • registry_custom.py registers the simulation environment of SUMO to the gym library to realize the connection with GRL algorithms.
  • specific_environment.py defines the elements in MDPs, including state representation, action space and reward function.
  • Experiment folder is the core program of co-simulation under different GRL algorithms, including the initialization of the simulation environment, the initialization of the neural network, the training and testing of GRL algorithms, and the preservation of the training and testing results.
  • GRL_Trained_Models folder stores the trained GRL model when the training process ends.

Tutorial

You can simply run "main.py" in Pycharm to simulate the GRL algorithm, and observe the simulation process in SUMO platform. You can generate training plot such as Reward curve:

Verification of other algorithms

If you want to verify other algorithms, you can develop the source code as needed under the "Experiment folder", and don't forget to change the imported python script in "main.py". In addition, you can also construct your own network in GRLNet folder.

Verification of other traffic scenario

If you want to verify other traffic scenario, you can define a new scenario in "network.py". You can refer to the documentation of SUMO for more details .

Owner
XXQQ
XXQQ
CO-PILOT: COllaborative Planning and reInforcement Learning On sub-Task curriculum

CO-PILOT CO-PILOT: COllaborative Planning and reInforcement Learning On sub-Task curriculum, NeurIPS 2021, Shuang Ao, Tianyi Zhou, Guodong Long, Qingh

Shuang Ao 1 Feb 18, 2022
PyTorch implementation of ''Background Activation Suppression for Weakly Supervised Object Localization''.

Background Activation Suppression for Weakly Supervised Object Localization PyTorch implementation of ''Background Activation Suppression for Weakly S

35 Jan 06, 2023
Chess reinforcement learning by AlphaGo Zero methods.

About Chess reinforcement learning by AlphaGo Zero methods. This project is based on these main resources: DeepMind's Oct 19th publication: Mastering

Samuel 2k Dec 29, 2022
Pytorch Implementation of LNSNet for Superpixel Segmentation

LNSNet Overview Official implementation of Learning the Superpixel in a Non-iterative and Lifelong Manner (CVPR'21) Learning Strategy The proposed LNS

42 Oct 11, 2022
Code for "Reconstructing 3D Human Pose by Watching Humans in the Mirror", CVPR 2021 oral

Reconstructing 3D Human Pose by Watching Humans in the Mirror Qi Fang*, Qing Shuai*, Junting Dong, Hujun Bao, Xiaowei Zhou CVPR 2021 Oral The videos a

ZJU3DV 178 Dec 13, 2022
Categorizing comments on YouTube into different categories.

Youtube Comments Categorization This repo is for categorizing comments on a youtube video into different categories. negative (grievances, complaints,

Rhitik 5 Nov 26, 2022
YOLOv5 detection interface - PyQt5 implementation

所有代码已上传,直接clone后,运行yolo_win.py即可开启界面。 2021/9/29:加入置信度选择 界面是在ultralytics的yolov5基础上建立的,界面使用pyqt5实现,内容较简单,娱乐而已。 功能: 模型选择 本地文件选择(视频图片均可) 开关摄像头

487 Dec 27, 2022
Pytorch implementation of "Training a 85.4% Top-1 Accuracy Vision Transformer with 56M Parameters on ImageNet"

Token Labeling: Training an 85.4% Top-1 Accuracy Vision Transformer with 56M Parameters on ImageNet (arxiv) This is a Pytorch implementation of our te

蒋子航 383 Dec 27, 2022
Motion Reconstruction Code and Data for Skills from Videos (SFV)

Motion Reconstruction Code and Data for Skills from Videos (SFV) This repo contains the data and the code for motion reconstruction component of the S

268 Dec 01, 2022
Meta Learning Backpropagation And Improving It (VSML)

Meta Learning Backpropagation And Improving It (VSML) This is research code for the NeurIPS 2021 publication Kirsch & Schmidhuber 2021. Many concepts

Louis Kirsch 22 Dec 21, 2022
[ECCV'20] Convolutional Occupancy Networks

Convolutional Occupancy Networks Paper | Supplementary | Video | Teaser Video | Project Page | Blog Post This repository contains the implementation o

622 Dec 30, 2022
Official implementation of "A Shared Representation for Photorealistic Driving Simulators" in PyTorch.

A Shared Representation for Photorealistic Driving Simulators The official code for the paper: "A Shared Representation for Photorealistic Driving Sim

VITA lab at EPFL 7 Oct 13, 2022
A PyTorch implementation of "DGC-Net: Dense Geometric Correspondence Network"

DGC-Net: Dense Geometric Correspondence Network This is a PyTorch implementation of our work "DGC-Net: Dense Geometric Correspondence Network" TL;DR A

191 Dec 16, 2022
DeepHawkeye is a library to detect unusual patterns in images using features from pretrained neural networks

English | 简体中文 Introduction DeepHawkeye is a library to detect unusual patterns in images using features from pretrained neural networks Reference Pat

CV Newbie 28 Dec 13, 2022
Codes and pretrained weights for winning submission of 2021 Brain Tumor Segmentation (BraTS) Challenge

Winning submission to the 2021 Brain Tumor Segmentation Challenge This repo contains the codes and pretrained weights for the winning submission to th

94 Dec 28, 2022
Solving reinforcement learning tasks which require language and vision

Multimodal Reinforcement Learning JAX implementations of the following multimodal reinforcement learning approaches. Dual-coding Episodic Memory from

Henry Prior 31 Feb 26, 2022
A curated list of awesome Model-Based RL resources

Awesome Model-Based Reinforcement Learning This is a collection of research papers for model-based reinforcement learning (mbrl). And the repository w

OpenDILab 427 Jan 03, 2023
The implementation of the paper "A Deep Feature Aggregation Network for Accurate Indoor Camera Localization".

A Deep Feature Aggregation Network for Accurate Indoor Camera Localization This is the PyTorch implementation of our paper "A Deep Feature Aggregation

9 Dec 09, 2022
VQGAN+CLIP Colab Notebook with user-friendly interface.

VQGAN+CLIP and other image generation system VQGAN+CLIP Colab Notebook with user-friendly interface. Latest Notebook: Mse regulized zquantize Notebook

Justin John 227 Jan 05, 2023
Towards Implicit Text-Guided 3D Shape Generation (CVPR2022)

Towards Implicit Text-Guided 3D Shape Generation Towards Implicit Text-Guided 3D Shape Generation (CVPR2022) Code for the paper [Towards Implicit Text

55 Dec 16, 2022