Code to generate datasets used in "How Useful is Self-Supervised Pretraining for Visual Tasks?"

Overview

Synthetic dataset rendering

Framework for producing the synthetic datasets used in:

How Useful is Self-Supervised Pretraining for Visual Tasks?
Alejandro Newell and Jia Deng. CVPR, 2020. arXiv:2003.14323

Experiment code can be found here.

This is a general purpose synthetic setting supporting single-object or multi-object images providing annotations for object classification, object pose estimation, segmentation, and depth estimation.

Setup

Download and set up Blender 2.80 (this code has not been tested on more recent Blender versions).

Blender uses its own Python, to which we need to add an extra package. In the Blender installation, find the python directory and run:

cd path/to/blender/2.80/python/bin
./python3.7m -m ensure pip
./pip3 install gin_config

For distributed rendering and additional dataset prep, use your own Python installation (not the Blender version). Everything was tested with Python 3.7 and the following extra packages:

sudo apt install libopenexr-dev
pip install ray ray[tune] h5py openexr scikit-image

External data

Download ShapeNetCore.v2 and DTD.

By default, it is assumed external datasets will be placed in syn_benchmark/datasets (e.g. syn_benchmark/datasets/ShapeNetCore.v2). If this is not the case, change any paths as necessary in paths.py.

Dataset Generation

Try a test run with:

blender --background --python render.py -- -d test_dataset

The argument -d, --dataset_name specifies the output directory which will be placed in the directory defined by pahs.DATA_DIR. Dataset settings can be modified either by selecting a gin config file (-g) or by modifying parameters (-p), for example:

blender --background --python render.py -- -g render_multi
blender --background --python render.py -- -p "material.use_texture = False" "object.random_viewpoint = 0"
blender --background --python render.py -- -g render_multi -p "batch.num_samples = 100"

Manual arguments passed in through -p will override those in the provided gin file. Please check out config/render_single.gin to see what options can be modified.

Distributed rendering

To scale up dataset creation, rendering is split into smaller jobs that can be sent out to individual workers for parallelization on a single machine or on a cluster. The library Ray is used to manage workers automatically. This allows large-scale distributed, parallel processes which are easy to restart in case anything crashes.

Calling python distributed_render.py will by default produce small versions of the 12 single-object datasets used in the paper. Arguments are available to control the overall dataset size and to interface with Ray. The script can be modified as needed to produce individual datasets or to modify dataset properties (e.g. texture, lighting, etc).

To produce multi-object images with depth and segmentation ground truth, add the argument --is_multi.

Further processing

After running the rendering script, you will be left with a large number of individual files containing rendered images and metadata pertaining to class labels and other scene information. Before running the main experiment code it is important that this data is preprocessed.

There are two key steps:

  • consolidation of raw data to HDF5 datasets: python preprocess_data.py -d test_dataset -f
  • image resizing and preprocessing: python preprocess_data.py -d test_dataset -p

If working with EXR images produced for segmentation/depth data make sure to add the argument -e.

-f, --to_hdf5: The first step will move all image files and metadata into HDF5 dataset files.

An important step that occurs here is conversion of EXR data to PNG data. The EXR output from Blender contains both the rendered image and corresponding depth, instance segmentation, and semantic segmentation data. After running this script, the rendered image is stored as one PNG and the depth and segmentation channels are concatenated into another PNG image.

After this step, I recommend removing the original small files if disk space is a concern, all raw data is fully preserved in the img_XX.h5 files. Note, the data is stored as an encoded PNG, if you want to read the image into Python you can do the following:

f = h5py.File('path/to/your/dataset/imgs_00.h5', 'r')
img_idx = 0
png_data = f['png_images'][img_idx]

img = imageio.imread(io.BytesIO(png_data))
# or alternatively
img = util.img_read_and_resize(png_data)

-p, --preprocess: Once the raw data has been moved into HDF5 files, it can be quickly processed for use in experiments. This preprocessing simply takes care of steps that would otherwise be performed over and over again during training such as image resizing and normalization. One of the more expensive steps that is taken care of here is conversion to LAB color space.

This preprocessing step prepares a single HDF5 file which ready to be used with the experiment code. Unlike the files created in the previous step, this data has been processed and some information may be lost from the original images especially if they have been resized to a lower resolution.

Owner
Princeton Vision & Learning Lab
Princeton Vision & Learning Lab
Cockpit is a visual and statistical debugger specifically designed for deep learning.

Cockpit: A Practical Debugging Tool for Training Deep Neural Networks

Felix Dangel 421 Dec 29, 2022
Implementation of a Transformer that Ponders, using the scheme from the PonderNet paper

Ponder(ing) Transformer Implementation of a Transformer that learns to adapt the number of computational steps it takes depending on the difficulty of

Phil Wang 65 Oct 04, 2022
YoloAll is a collection of yolo all versions. you you use YoloAll to test yolov3/yolov5/yolox/yolo_fastest

官方讨论群 QQ群:552703875 微信群:15158106211(先加作者微信,再邀请入群) YoloAll项目简介 YoloAll是一个将当前主流Yolo版本集成到同一个UI界面下的推理预测工具。可以迅速切换不同的yolo版本,并且可以针对图片,视频,摄像头码流进行实时推理,可以很方便,直观

DL-Practise 244 Jan 01, 2023
Multi Camera Calibration

Multi Camera Calibration 'modules/camera_calibration/app/camera_calibration.cpp' is for calculating extrinsic parameter of each individual cameras. 'm

7 Dec 01, 2022
Explaining neural decisions contrastively to alternative decisions.

Contrastive Explanations for Model Interpretability This is the repository for the paper "Contrastive Explanations for Model Interpretability", about

AI2 16 Oct 16, 2022
Simple sinc interpolation in PyTorch.

Kazane: simple sinc interpolation for 1D signal in PyTorch Kazane utilize FFT based convolution to provide fast sinc interpolation for 1D signal when

Chin-Yun Yu 10 May 03, 2022
Stochastic gradient descent with model building

Stochastic Model Building (SMB) This repository includes a new fast and robust stochastic optimization algorithm for training deep learning models. Th

S. Ilker Birbil 22 Jan 19, 2022
Keras implementation of Normalizer-Free Networks and SGD - Adaptive Gradient Clipping

Keras implementation of Normalizer-Free Networks and SGD - Adaptive Gradient Clipping

Yam Peleg 63 Sep 21, 2022
This program writes christmas wish programmatically. It is using turtle as a pen pointer draw christmas trees and stars.

Introduction This is a simple program is written in python and turtle library. The objective of this program is to wish merry Christmas programmatical

Gunarakulan Gunaretnam 1 Dec 25, 2021
AirLoop: Lifelong Loop Closure Detection

AirLoop This repo contains the source code for paper: Dasong Gao, Chen Wang, Sebastian Scherer. "AirLoop: Lifelong Loop Closure Detection." arXiv prep

Chen Wang 53 Jan 03, 2023
Opinionated code formatter, just like Python's black code formatter but for Beancount

beancount-black Opinionated code formatter, just like Python's black code formatter but for Beancount Try it out online here Features MIT licensed - b

Launch Platform 16 Oct 11, 2022
Python implementation of 3D facial mesh exaggeration using the techniques described in the paper: Computational Caricaturization of Surfaces.

Python implementation of 3D facial mesh exaggeration using the techniques described in the paper: Computational Caricaturization of Surfaces.

Wonjong Jang 8 Nov 01, 2022
Gym environment for FLIPIT: The Game of "Stealthy Takeover"

gym-flipit Gym environment for FLIPIT: The Game of "Stealthy Takeover" invented by Marten van Dijk, Ari Juels, Alina Oprea, and Ronald L. Rivest. Desi

Lisa Oakley 2 Dec 15, 2021
[CIKM 2021] Enhancing Aspect-Based Sentiment Analysis with Supervised Contrastive Learning

Enhancing Aspect-Based Sentiment Analysis with Supervised Contrastive Learning. This repo contains the PyTorch code and implementation for the paper E

Akuchi 18 Dec 22, 2022
This is a file about Unet implemented in Pytorch

Unet this is an implemetion of Unet in Pytorch and it's architecture is as follows which is the same with paper of Unet component of Unet Convolution

Dragon 1 Dec 03, 2021
UCSD Oasis platform

oasis UCSD Oasis platform Local project setup Install Docker Compose and make sure you have Pip installed Clone the project and go to the project fold

InSTEDD 4 Jun 16, 2021
Record radiologists' eye gaze when they are labeling images.

Record radiologists' eye gaze when they are labeling images. Read for installation, usage, and deep learning examples. Why use MicEye Versatile As a l

24 Nov 03, 2022
[TIP2020] Adaptive Graph Representation Learning for Video Person Re-identification

Introduction This is the PyTorch implementation for Adaptive Graph Representation Learning for Video Person Re-identification. Get started git clone h

WuYiming 41 Dec 12, 2022
A PyTorch implementation of EventProp [https://arxiv.org/abs/2009.08378], a method to train Spiking Neural Networks

Spiking Neural Network training with EventProp This is an unofficial PyTorch implemenation of EventProp, a method to compute exact gradients for Spiki

Pedro Savarese 35 Jul 29, 2022
Conflict-aware Inference of Python Compatible Runtime Environments with Domain Knowledge Graph, ICSE 2022

PyCRE Conflict-aware Inference of Python Compatible Runtime Environments with Domain Knowledge Graph, ICSE 2022 Dependencies This project is developed

<a href=[email protected]"> 7 May 06, 2022