Training code and evaluation benchmarks for the "Self-Supervised Policy Adaptation during Deployment" paper.

Last update: Nov 01, 2022

Overview

Self-Supervised Policy Adaptation during Deployment

PyTorch implementation of PAD and evaluation benchmarks from

Self-Supervised Policy Adaptation during Deployment

Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Alenyà, Pieter Abbeel, Alexei A. Efros, Lerrel Pinto, Xiaolong Wang

[Paper] [Website]

Citation

If you find our work useful in your research, please consider citing the paper as follows:

@article{hansen2020deployment,
  title={Self-Supervised Policy Adaptation during Deployment},
  author={Nicklas Hansen and Rishabh Jangir and Yu Sun and Guillem Alenyà and Pieter Abbeel and Alexei A. Efros and Lerrel Pinto and Xiaolong Wang},
  year={2020},
  eprint={2007.04309},
  archivePrefix={arXiv},
  primaryClass={cs.LG}
}

Setup

We assume that you have access to a GPU with CUDA >=9.2 support. All dependencies can then be installed with the following commands:

conda env create -f setup/conda.yml
conda activate pad
sh setup/install_envs.sh

Training & Evaluation

We have prepared training and evaluation scripts that can be run by sh scripts/train.sh and sh scripts/eval.sh. Alternatively, you can call the python scripts directly, e.g. for training call

CUDA_VISIBLE_DEVICES=0 python3 src/train.py \
    --domain_name cartpole \
    --task_name swingup \
    --action_repeat 8 \
    --mode train \
    --use_inv \
    --num_shared_layers 8 \
    --seed 0 \
    --work_dir logs/cartpole_swingup/inv/0 \
    --save_model

which should give you an output of the form

| train | E: 1 | S: 1000 | D: 0.8 s | R: 0.0000 | BR: 0.0000 | 
  ALOSS: 0.0000 | CLOSS: 0.0000 | RLOSS: 0.0000

We provide a pre-trained model that can be used for evaluation. To run Policy Adaptation during Deployment, call

CUDA_VISIBLE_DEVICES=0 python3 src/eval.py \
    --domain_name cartpole \
    --task_name swingup \
    --action_repeat 8 \
    --mode color_hard \
    --use_inv \
    --num_shared_layers 8 \
    --seed 0 \
    --work_dir logs/cartpole_swingup/inv/0 \
    --pad_checkpoint 500k

which should give you an output of the form

Evaluating logs/cartpole_swingup/inv/0 for 100 episodes (mode: color_hard)
eval reward: 666

Policy Adaptation during Deployment of logs/cartpole_swingup/inv/0 for 100 episodes (mode: color_hard)
pad reward: 722

Here's a few samples from the training and test environments of our benchmark:

Please refer to the project page and paper for results and experimental details.

Acknowledgements

We want to thank the numerous researchers and engineers involved in work of which this implementation is based on. Our SAC implementation is based on this repository, the original DeepMind Control suite is available here and the gym wrapper for it is available here. Go check them out!

Training code and evaluation benchmarks for the "Self-Supervised Policy Adaptation during Deployment" paper.

Related tags

Overview

Self-Supervised Policy Adaptation during Deployment

Citation

Setup

Training & Evaluation

Acknowledgements

Owner

Nicklas Hansen

Interpretation of T cell states using reference single-cell atlases

Source code for "Roto-translated Local Coordinate Framesfor Interacting Dynamical Systems"

A deep learning tabular classification architecture inspired by TabTransformer with integrated gated multilayer perceptron.

Global-Local Context Network for Person Search

MAVE: : A Product Dataset for Multi-source Attribute Value Extraction

Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view Geometry Consistency[ECCV 2020]

Official implementation of "UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspective with Transformer"

Official implementation of the ICCV 2021 paper: "The Power of Points for Modeling Humans in Clothing".

🐾 Semantic segmentation of paws from cute pet images (PyTorch)

Space-event-trace - Tracing service for spaceteam events

Final project code: Implementing BicycleGAN, for CIS680 FA21 at University of Pennsylvania

The personal repository of the work: DanceNet3D: Music Based Dance Generation with Parametric Motion Transformer.

Single Red Blood Cell Hydrodynamic Traps Via the Generative Design

SwinIR: Image Restoration Using Swin Transformer

AAAI 2022 paper - Unifying Model Explainability and Robustness for Joint Text Classification and Rationale Extraction

Recurrent Conditional Query Learning

List of content farm sites like g.penzai.com.

[ICLR 2021, Spotlight] Large Scale Image Completion via Co-Modulated Generative Adversarial Networks

WaveFake: A Data Set to Facilitate Audio DeepFake Detection

TriMap: Large-scale Dimensionality Reduction Using Triplets

Training code and evaluation benchmarks for the "Self-Supervised Policy Adaptation during Deployment" paper.

Related tags

Overview

Self-Supervised Policy Adaptation during Deployment

Citation

Setup

Training & Evaluation

Acknowledgements

Owner

Nicklas Hansen

Interpretation of T cell states using reference single-cell atlases

Source code for "Roto-translated Local Coordinate Framesfor Interacting Dynamical Systems"

A deep learning tabular classification architecture inspired by TabTransformer with integrated gated multilayer perceptron.

Global-Local Context Network for Person Search

MAVE: : A Product Dataset for Multi-source Attribute Value Extraction

Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view Geometry Consistency[ECCV 2020]

Official implementation of "UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspective with Transformer"

Official implementation of the ICCV 2021 paper: "The Power of Points for Modeling Humans in Clothing".

🐾 Semantic segmentation of paws from cute pet images (PyTorch)

Space-event-trace - Tracing service for spaceteam events

Final project code: Implementing BicycleGAN, for CIS680 FA21 at University of Pennsylvania

The personal repository of the work: *DanceNet3D: Music Based Dance Generation with Parametric Motion Transformer*.

Single Red Blood Cell Hydrodynamic Traps Via the Generative Design

SwinIR: Image Restoration Using Swin Transformer

AAAI 2022 paper - Unifying Model Explainability and Robustness for Joint Text Classification and Rationale Extraction

Recurrent Conditional Query Learning

List of content farm sites like g.penzai.com.

[ICLR 2021, Spotlight] Large Scale Image Completion via Co-Modulated Generative Adversarial Networks

WaveFake: A Data Set to Facilitate Audio DeepFake Detection

TriMap: Large-scale Dimensionality Reduction Using Triplets

The personal repository of the work: DanceNet3D: Music Based Dance Generation with Parametric Motion Transformer.