A custom-designed Spider Robot trained to walk using Deep RL in a PyBullet Simulation

Overview

SpiderBot_DeepRL

Title: Implementation of Single and Multi-Agent Deep Reinforcement Learning Algorithms for a Walking Spider Robot Authors(s): Arijit Dasgupta, Chong Yu Quan

Welcome to our project! For this project, we aim to take our SpiderBot and make it walk using deep reinforcement learning. The code is written entirely in Python 3.7.7 and the following Python libraries are required for our code to work.

pybullet==3.0.6
numpy==1.18.5
matplotlib==3.3.2
tensorflow_probability==0.11.1
seaborn==0.11.0
pandas==1.1.4
tensorflow==2.3.1

Other than this, no additional software is needed for the code to work. The PyBullet Physics Engine is used for simulation using an OpenGL GUI. In this code, we have the following : -

  • A requirement.txt for required python libraries
  • SolidWorks CADs of the SpiderBot
  • SpiderBot URDFs for the SpiderBot
  • Folders for Training Logs & Plots
  • Two saved models of the SpiderBot Agent
  • Source Code for the Deep RL Implementation
  • Training Code to train the SpiderBot with Deep RL
  • Validation Code to test trained models
  • Postprocessing Code to generate plots of training

The code supports the following 5 algorithms (with their characteristics defined):

Algorithm Agent (Actor) Policy Learning Network Actions per Time-Step Action Space State Space
MAD3QN Multiple (Decentralised) Decentralised Separate Multiple Discrete Continuous
MAA2C Multiple (Decentralised) Decentralised Separate Multiple Discrete Continuous
A2CMA Single (Centralised) Decentralised Hybrid Multiple Discrete Continuous
A2CSA Single (Centralised) Centralised Hybrid Single Discrete Continuous
DDPG Single (Centralised) Centralised Separate Multiple Continuous Continuous

We will now walk through the folders and files.

Folders

SpiderBot_CADs

This folder contains all the part and assembly files for the SpiderBot. There are options for 3-legged, 4-legged, 6-legged & 8-legged SpiderBots.

SpiderBot_URDFs

This folder contains all URDF files and associated STL files for the SpiderBot. There are options for 3-legged, 4-legged, 6-legged & 8-legged SpiderBots.

Training_Logs & Training_Plots

Folders to store csv file of training data and PDF plots of training.

Saved_Models

Contains two saved models using DDPG. The FullyTrained Model (375 episodes) is able to walk well and up to 9 metres in the forward direction. The PartiallyTrained Model (50 episodes) can move forward slightly but only to a certain extent.

Source Code

SpiderBot_Environment.py

This file has the p_gym class. This uses pybullet and loads the plane environment (no obstacles) and the SpiderBot into the physics engine. The code allows an agent to retrieve state observations for a leg or whole SpiderBot and set a target velocity for joints in the SpiderBot. Finally, the code uses information from the physics engine to determine rewards for a time step.

SpiderBot_Neural_Network.py

This file has the classes for the fully-connected neural networks used. The Tensorflow 2 API is used to develop the neural networks. Depending on the algorithm and number of SpiderBot legs, the neural networks are customised for them. There is all a call method to do a forward propagation through the neural network.

SpiderBot_Agent.py

This file is a long one, which has all the operations of the agent for all 5 algorithms. It initialises the neural networks based on the algorithm in the constructor. The class also has the functionality to update the target networks for DDPG & MAD3QN. Additionally, it has a long list of methods to apply gradients for each one of the algorithms. In these methods, the TensorFlow 2 computational graph and gradient tapes are used to help in backpropagating the loss function. Finally the class also has the functionality to save all models and load all models.

SpiderBot_Replay_Buffer.py

This file contains the replay_buffer class that handles experience replay storage and operations like logging and sampling with a batch size.

SpiderBot_Walk.py

This file contains the walk function that is actually responsible for handling all training operations. This is where all the classes interact with each other. The episodes are looped through and the SpiderBot is trained. The training-related data is logged and saved as a csv into the Training_Logs folder while the best models are saved to the Saved_Models folder during training.

SpiderBot_Postprocessing.py

This file handles the plotting post-processing operations that takes the CSV file from the Training_Logs folder and saves the plot into the Training_Plots folder.

Main Code

SpiderBot_Train_Model.py

This file allows the user to set up the training session. In this file, the user can set 3 levels of configuration for training. The general config section has options for choosing algorithms, number of legs, target location, episodes etc. The Hyperparameters config section handles all hyperparameters of the entire training process. The reward structure config provides options for all the scalar rewards. The user must set all of these configs and run the file to train the SpiderBot. TIP: not using a GUI is faster for training, especially if you use a CUDA-enabled NVIDIA GPU.

SpiderBot_Validation.py

This file allows the user to validate and test a trained model, specially made for the Professors and TAs of SpiderBot to visualise our fully trained model.

How to train a model?

Unzip the SpiderBot_URDFS.zip file into the same directory. Open up SpiderBot_Train_Model.py for editing. The most important parameter is training_name that you must define. This is unique to a particular training session and all saved models, logs and plots are based on this training_name. After that set up your General Config:

#~~~~~~~~~~~~~~~~~~~~~~~~~~~ GENERAL CONFIG ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~#
training_name = "insert_training_name_here"
model = "DDPG"
num_of_legs = 8 
episodes = 375
target_location = 3
use_GUI = True
do_post_process = True
save_best_model = True
save_data = True

Following that, set up the configurations for the hyperparameters:

#~~~~~~~~~~~~~~~~~~~~~~~~~~~ HYPERPARAMETER CONFIG ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~#
time_step_size = 120./240
upper_angle = 60
lower_angle = -60
lr_actor = 0.00005
lr_critic = 0.0001
discount_rate = 0.9
update_target = None
tau = 0.005
max_mem_size = 1000000
batch_size = 512
max_action = 10
min_action = -10
noise = 1
epsilon = 1
epsilon_decay = 0.0001
epsilon_min = 0.01

Finally, set up the configuration for the reward structure:

#~~~~~~~~~~~~~~~~~~~~~~~~~~~ REWARD STRUCTURE CONFIG ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~#
forward_motion_reward = 500
forward_distance_reward = 250
sideways_velocity_punishment = 500
sideways_distance_penalty = 250
time_step_penalty = 1
flipped_penalty = 500
goal_reward = 500
out_of_range_penalty = 500

Then run the python code

> python SpiderBot_Train_Model.py

How to Validate/Test our Models?

To test the fully trained model, just run SpiderBot_Validation.py.

> python SpiderBot_Validation.py

If you wish to run the other saved model, the partially trained one, you can open up SpiderBot_Validation.py and edit the training_name from DDPG_FullyTrained to DDPG_PartiallyTrained in the config section as shown:

#~~~~~~~~~~~~ VALIDATION CONFIG SETUP ~~~~~~~~~~~~#
training_name = "DDPG_PartiallyTrained"
model = "DDPG"
target_location = 8
episodes = 100000000000 # A large number is set to put the simulation on loop

Video Demonstration

Owner
Arijit Dasgupta
Arijit Dasgupta
Explaining neural decisions contrastively to alternative decisions.

Contrastive Explanations for Model Interpretability This is the repository for the paper "Contrastive Explanations for Model Interpretability", about

AI2 16 Oct 16, 2022
PyTorch implementation of the wavelet analysis from Torrence & Compo

Continuous Wavelet Transforms in PyTorch This is a PyTorch implementation for the wavelet analysis outlined in Torrence and Compo (BAMS, 1998). The co

Tom Runia 262 Dec 21, 2022
Skyformer: Remodel Self-Attention with Gaussian Kernel and Nystr\"om Method (NeurIPS 2021)

Skyformer This repository is the official implementation of Skyformer: Remodel Self-Attention with Gaussian Kernel and Nystr"om Method (NeurIPS 2021).

Qi Zeng 46 Sep 20, 2022
A benchmark dataset for mesh multi-label-classification based on cube engravings introduced in MeshCNN

Double Cube Engravings This script creates a dataset for multi-label mesh clasification, with an intentionally difficult setup for point cloud classif

Yotam Erel 1 Nov 30, 2021
We present a regularized self-labeling approach to improve the generalization and robustness properties of fine-tuning.

Overview This repository provides the implementation for the paper "Improved Regularization and Robustness for Fine-tuning in Neural Networks", which

NEU-StatsML-Research 21 Sep 08, 2022
This repository contains PyTorch code for Robust Vision Transformers.

This repository contains PyTorch code for Robust Vision Transformers.

117 Dec 07, 2022
[NeurIPS 2020] This project provides a strong single-stage baseline for Long-Tailed Classification, Detection, and Instance Segmentation (LVIS).

A Strong Single-Stage Baseline for Long-Tailed Problems This project provides a strong single-stage baseline for Long-Tailed Classification (under Ima

Kaihua Tang 514 Dec 23, 2022
Pytorch implementation of Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors

Make-A-Scene - PyTorch Pytorch implementation (inofficial) of Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (https://arxiv.org/

Casual GAN Papers 259 Dec 28, 2022
The easiest way to use deep metric learning in your application. Modular, flexible, and extensible. Written in PyTorch.

News December 27: v1.1.0 New loss functions: CentroidTripletLoss and VICRegLoss Mean reciprocal rank + per-class accuracies See the release notes Than

Kevin Musgrave 5k Jan 05, 2023
Jittor implementation of Recursive-NeRF: An Efficient and Dynamically Growing NeRF

Recursive-NeRF: An Efficient and Dynamically Growing NeRF This is a Jittor implementation of Recursive-NeRF: An Efficient and Dynamically Growing NeRF

33 Nov 30, 2022
PyTorch code for the paper "Complementarity is the King: Multi-modal and Multi-grained Hierarchical Semantic Enhancement Network for Cross-modal Retrieval".

Complementarity is the King: Multi-modal and Multi-grained Hierarchical Semantic Enhancement Network for Cross-modal Retrieval (M2HSE) PyTorch code fo

Xinlei-Pei 6 Dec 23, 2022
Tensorflow solution of NER task Using BiLSTM-CRF model with Google BERT Fine-tuning And private Server services

Tensorflow solution of NER task Using BiLSTM-CRF model with Google BERT Fine-tuning

MaCan 4.2k Dec 29, 2022
MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution Shifts and Training Conflicts (ICLR 2022)

MetaShift: A Dataset of Datasets for Evaluating Distribution Shifts and Training Conflicts This repo provides the PyTorch source code of our paper: Me

88 Jan 04, 2023
IJON is an annotation mechanism that analysts can use to guide fuzzers such as AFL.

IJON SPACE EXPLORER IJON is an annotation mechanism that analysts can use to guide fuzzers such as AFL. Using only a small (usually one line) annotati

Chair for Sys­tems Se­cu­ri­ty 146 Dec 16, 2022
Planning from Pixels in Environments with Combinatorially Hard Search Spaces -- NeurIPS 2021

PPGS: Planning from Pixels in Environments with Combinatorially Hard Search Spaces Environment Setup We recommend pipenv for creating and managing vir

Autonomous Learning Group 11 Jun 26, 2022
Python wrapper class for OpenVINO Model Server. User can submit inference request to OVMS with just a few lines of code

Python wrapper class for OpenVINO Model Server. User can submit inference request to OVMS with just a few lines of code.

Yasunori Shimura 7 Jul 27, 2022
Implementation of ICCV21 paper: PnP-DETR: Towards Efficient Visual Analysis with Transformers

Implementation of ICCV 2021 paper: PnP-DETR: Towards Efficient Visual Analysis with Transformers arxiv This repository is based on detr Recently, DETR

twang 113 Dec 27, 2022
MonoRec: Semi-Supervised Dense Reconstruction in Dynamic Environments from a Single Moving Camera

MonoRec: Semi-Supervised Dense Reconstruction in Dynamic Environments from a Single Moving Camera

Felix Wimbauer 494 Jan 06, 2023
Official implementation of the paper "Lightweight Deep CNN for Natural Image Matting via Similarity Preserving Knowledge Distillation"

Lightweight-Deep-CNN-for-Natural-Image-Matting-via-Similarity-Preserving-Knowledge-Distillation Introduction Accepted at IEEE Signal Processing Letter

DongGeun-Yoon 19 Jun 07, 2022
Implementation for paper LadderNet: Multi-path networks based on U-Net for medical image segmentation

Implementation for paper LadderNet: Multi-path networks based on U-Net for medical image segmentation This implementation is based on orobix implement

Juntang Zhuang 116 Sep 06, 2022