Sync2Gen Code for ICCV 2021 paper: Scene Synthesis via Uncertainty-Driven Attribute Synchronization

Last update: Dec 30, 2022

Related tags

Deep Learning Sync2Gen

Overview

Sync2Gen

Code for ICCV 2021 paper: Scene Synthesis via Uncertainty-Driven Attribute Synchronization

0. Environment

Environment: python 3.6 and cuda 10.0 on Ubuntu 18.04

Pytorch 1.4.0
tensorflow 1.14.0 (for tensorboard)

1. Dataset

├──dataset_3dfront/
    ├──data
        ├── bedroom
            ├── 0_abs.npy
            ├── 0_rel.pkl
            ├── ...
        ├── living
            ├── 0_abs.npy
            ├── 0_rel.pkl
            ├── ...
        ├── train_bedroom.txt
        ├── train_living.txt
        ├── val_bedroom.txt
        └── val_living.txt

See 3D-FRONT Dataset for dataset generation.

2. VAE

2.1 Generate scenes from random noises

Download the pretrained model from https://drive.google.com/file/d/1VKNlEdUj1RBUOjBaBxE5xQvfsZodVjam/view?usp=sharing

Sync2Gen
└── log
    └── 3dfront
        ├── bedroom
        │   └── vaef_lr0001_w00001_B64
        │       ├── checkpoint_eval799.tar
        │       └── pairs
        └── living
            └── vaef_lr0001_w00001_B64
                ├── checkpoint_eval799.tar
                └── pairs

type='bedroom'; # or living
CUDA_VISIBLE_DEVICES=0 python ./test_sparse.py  --type $type  --log_dir ./log/3dfront/$type/vaef_lr0001_w00001_B64 --model_dict=model_scene_forward --max_parts=80 --num_class=20 --num_each_class=4 --batch_size=32 --variational --latent_dim 20 --abs_dim 16  --weight_kld 0.0001  --learning_rate 0.001 --use_dumped_pairs --dump_results --gen_from_noise --num_gen_from_noise 100

The predictions are dumped in ./dump/$type/vaef_lr0001_w00001_B64

2.2 Training

To train the network:

type='bedroom'; # or living
CUDA_VISIBLE_DEVICES=0 python ./train_sparse.py --data_path ./dataset_3dfront/data  --type $type  --log_dir ./log/3dfront/$type/vaef_lr0001_w00001_B64  --model_dict=model_scene_forward --max_parts=80 --num_class=20 --num_each_class=4 --batch_size=64 --variational --latent_dim 20 --abs_dim 16  --weight_kld 0.0001  --learning_rate 0.001

3. Bayesian optimization

cd optimization

3.1 Prior generation

See Prior generation.

3.2 Optimization

type=bedroom # or living;
bash opt.sh $type vaef_lr0001_w00001_B64  EXP_NAME

We use Pytorch-LBFGS for optimization.

3.3 Visualization

There is a simple visualization tool:

type=bedroom # or living
bash vis.sh $type vaef_lr0001_w00001_B64 EXP_NAME

The visualization is in ./vis. {i:04d}_2(3)d_pred.png is the initial prediction from VAE. {i:04d}_2(3)d_sync.png is the optimized layout after synchronization.

Acknowledgements

The repo is built based on:

We thank the authors for their great job.

Contact

If you have any questions, you can contact Haitao Yang (yanghtr [AT] outlook [DOT] com).

Sync2Gen Code for ICCV 2021 paper: Scene Synthesis via Uncertainty-Driven Attribute Synchronization

Related tags

Overview

Sync2Gen

0. Environment

1. Dataset

2. VAE

2.1 Generate scenes from random noises

2.2 Training

3. Bayesian optimization

3.1 Prior generation

3.2 Optimization

3.3 Visualization

Acknowledgements

Contact

Owner

Haitao Yang

MixRNet(Using mixup as regularization and tuning hyper-parameters for ResNets)

Demo code for paper "Learning optical flow from still images", CVPR 2021.

This repository contains numerical implementation for the paper Intertemporal Pricing under Reference Effects: Integrating Reference Effects and Consumer Heterogeneity.

This is the source code of the 1st place solution for segmentation task (with Dice 90.32%) in 2021 CCF BDCI challenge.

PyQt6 configuration in yaml format providing the most simple script.

A object detecting neural network powered by the yolo architecture and leveraging the PyTorch framework and associated libraries.

CowHerd is a partially-observed reinforcement learning environment

Official Implementation of SimIPU: Simple 2D Image and 3D Point Cloud Unsupervised Pre-Training for Spatial-Aware Visual Representations

Official implementation for CVPR 2021 paper: Adaptive Class Suppression Loss for Long-Tail Object Detection

Data-Uncertainty Guided Multi-Phase Learning for Semi-supervised Object Detection

Google-drive-to-sqlite - Create a SQLite database containing metadata from Google Drive

Deep Learning: Architectures & Methods Project: Deep Learning for Audio Super-Resolution

This is project is the implementation of the DeepShift: Towards Multiplication-Less Neural Networks paper

Memory Defense: More Robust Classificationvia a Memory-Masking Autoencoder

Official implementation of ACTION-Net: Multipath Excitation for Action Recognition (CVPR'21).

A modular, open and non-proprietary toolkit for core robotic functionalities by harnessing deep learning

Server files for UltimateLabeling

:boar: :bear: Deep Learning based Python Library for Stock Market Prediction and Modelling

Winning solution of the Indoor Location & Navigation Kaggle competition

An MQA (Studio, originalSampleRate) identifier for lossless flac files written in Python.