Code release for our paper, "SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo"

Last update: Dec 14, 2022

Related tags

Overview

SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo

Thomas Kollar, Michael Laskey, Kevin Stone, Brijen Thananjeyan, Mark Tjersland

This repo contains the code to train the SimNet architecture on procedurally generated simulation data from scratch (no transfer learning required). We also provide a small set of in-house manually labelled validation data containing 3d oriented bounding box labels.

Training the model

Requirements

You will need a Nvidia GPU with at least 12GB of RAM. All code was tested and developed on Ubuntu 20.04.

All commands are assumed to be run from the root of the simnet repo directory (represented by $SIMNET_REPO in commands below).

Setup

Python

Create a python 3.8 virtual environment and install requirements:

cd $SIMNET_REPO
conda create -y --prefix ./env python=3.8
./env/bin/python -m pip install --upgrade pip
./env/bin/python -m pip install -r frozen_requirements.txt

Docker

Make sure docker is installed and working without requiring sudo. If it is not installed, follow the official instructions for setting it up.

docker ps

Wandb

Launch wandb local server for logging training results (you do not need to do this if you already have a wandb account setup). This will launch a local webserver http://localhost:8080 using docker that you can use to visualize training progress and validation images. You will have to visit the http://localhost:8080/authorize page to get the local API access token (this can take a few minutes the first time). Once you get the key you can paste it into the terminal to continue.

cd $SIMNET_REPO
./env/bin/wandb local

Datasets

Download and untar train+val datasets simnet2021a.tar (18GB, md5 checksum:b8e1d3cb7200b44b1de223e87141f14b). This file contains all the training and validation you need to replicate our small objects results.

cd $SIMNET_REPO
wget https://tri-robotics-public.s3.amazonaws.com/github/simnet/datasets/simnet2021a.tar -P datasets
tar xf datasets/simnet2021a.tar -C datasets

Train and Validate

Overfit test:

./runner.sh net_train.py @config/net_config_overfit.txt

Full training run (requires 12GB GPU memory)

./runner.sh net_train.py @config/net_config.txt

Results

Check wandb (http://localhost:8080) to see training progress. On a Titan V, it takes about 48 hours for training to converge, but decent validation results can be seen around 24 hours.

Example validation image visualization:

Example 3D oriented bounding box mAP on validation dataset:

Licenses

The source code is released under the MIT license.

The datasets are released under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Comments

depth noise model

I was looking through the code and was curious about the depth noise model. I found this: https://github.com/ToyotaResearchInstitute/simnet/blob/main/simnet/lib/camera.py but I can't seem to find camera_noise. Is it in the repository?

opened by seann999 1
Pre-trained Models

Hi Kevin and the team,

Thanks for making the data and code available, really impressive work on the paper.

Is there any plans to make the pre-trained model available, especially the SimNet benchmarked in the paper.

Thanks,

opened by ppyht2 0

Code related to "Have Your Text and Use It Too! End-to-End Neural Data-to-Text Generation with Semantic Fidelity" paper

DataTuner You have just found the DataTuner. This repository provides tools for fine-tuning language models for a task. See LICENSE.txt for license de

81 Jan 1, 2023

Code for CVPR2021 paper "Learning Salient Boundary Feature for Anchor-free Temporal Action Localization"

AFSD: Learning Salient Boundary Feature for Anchor-free Temporal Action Localization This is an official implementation in PyTorch of AFSD. Our paper

146 Dec 24, 2022

Code for the ACL2021 paper "Combining Static Word Embedding and Contextual Representations for Bilingual Lexicon Induction"

CSCBLI Code for our ACL Findings 2021 paper, "Combining Static Word Embedding and Contextual Representations for Bilingual Lexicon Induction". Require

12 Oct 8, 2022

This repository contains the code for the paper "SCANimate: Weakly Supervised Learning of Skinned Clothed Avatar Networks"

SCANimate: Weakly Supervised Learning of Skinned Clothed Avatar Networks (CVPR 2021 Oral) This repository contains the official PyTorch implementation

235 Dec 18, 2022

SceneCollisionNet This repo contains the code for "Object Rearrangement Using Learned Implicit Collision Functions", an ICRA 2021 paper. For more info

31 Nov 22, 2022

Dataset and Code for ICCV 2021 paper "Real-world Video Super-resolution: A Benchmark Dataset and A Decomposition based Learning Scheme"

Dataset and Code for RealVSR Real-world Video Super-resolution: A Benchmark Dataset and A Decomposition based Learning Scheme Xi Yang, Wangmeng Xiang,

91 Nov 22, 2022

Code release for our paper, "SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo"

Related tags

Overview

SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo

Training the model

Requirements

Setup

Python

Docker

Wandb

Datasets

Train and Validate

Results

Licenses

You might also like...

Code related to "Have Your Text and Use It Too! End-to-End Neural Data-to-Text Generation with Semantic Fidelity" paper

Code for CVPR2021 paper "Learning Salient Boundary Feature for Anchor-free Temporal Action Localization"

Code for the ACL2021 paper "Combining Static Word Embedding and Contextual Representations for Bilingual Lexicon Induction"

This repository contains the code for the paper "SCANimate: Weakly Supervised Learning of Skinned Clothed Avatar Networks"

SceneCollisionNet This repo contains the code for "Object Rearrangement Using Learned Implicit Collision Functions", an ICRA 2021 paper. For more info

Dataset and Code for ICCV 2021 paper "Real-world Video Super-resolution: A Benchmark Dataset and A Decomposition based Learning Scheme"

Code for paper "Role-based network embedding via structural features reconstruction with degree-regularized constraint"

Code for the paper: Fusformer: A Transformer-based Fusion Approach for Hyperspectral Image Super-resolution

The code for CVPR2022 paper "Likert Scoring with Grade Decoupling for Long-term Action Assessment".

Comments

depth noise model

Pre-trained Models

Releases(v0.0.1)

v0.0.1(Jul 19, 2021)

Owner

Handwritten Character Recognition using CNN

Code for the "Sensing leg movement enhances wearable monitoring of energy expenditure" paper.

Text page dewarping using a "cubic sheet" model

text detection mainly based on ctpn model in tensorflow, id card detect, connectionist text proposal network

Code for generating synthetic text images as described in "Synthetic Data for Text Localisation in Natural Images", Ankush Gupta, Andrea Vedaldi, Andrew Zisserman, CVPR 2016.

OCR of Chicago 1909 Renumbering Plan

Textboxes implementation with Tensorflow (python)

零样本学习测评基准，中文版

A python screen recorder for low-end computers, provides high quality video output.

A version of nrsc5-gui that merges the interface developed by cmnybo with the architecture developed by zefie in order to start a new baseline that is not heavily dependent upon Python processing.

CNN+Attention+Seq2Seq

Page to PAGE Layout Analysis Tool

A simple Digits Recogniser made in Python

Markup for note taking

Indonesian ID Card OCR using tesseract OCR

This is a pytorch re-implementation of EAST: An Efficient and Accurate Scene Text Detector.

Optical character recognition for Japanese text, with the main focus being Japanese manga

基于openpose和图像分类的手语识别项目

Maze generator and solver with python

OpenGait is a flexible and extensible gait recognition project