Official code release for: EditGAN: High-Precision Semantic Image Editing

Overview

EditGAN

Official code release for:

EditGAN: High-Precision Semantic Image Editing

Huan Ling*, Karsten Kreis*, Daiqing Li, Seung Wook Kim, Antonio Torralba, Sanja Fidler

(* authors contributed equally)

NeurIPS 2021

[project page] [paper] [supplementary material]

Demos and results

Left: The video showcases EditGAN in an interacitve demo tool. Right: The video demonstrates EditGAN where we apply multiple edits and exploit pre-defined editing vectors. Note that the demo is accelerated. See paper for run times.

Left: The video shows interpolations and combinations of multiple editing vectors. Right: The video presents the results of applying EditGAN editing vectors on out-of-domain images.

Requirements

  • Python 3.8 is supported.

  • Pytorch >= 1.4.0.

  • The code is tested with CUDA 10.1 toolkit with Pytorch==1.4.0 and CUDA 11.4 with Pytorch==1.10.0.

  • All results in our paper are based on NVIDIA Tesla V100 GPUs with 32GB memory.

  • Set up python environment:

virtualenv env
source env/bin/activate
pip install -r requirements.txt
  • Add the project to PYTHONPATH:
export PYTHONPATH=$PWD

Use of pre-trained model

We released a pre-trained model for the car class. Follow these steps to set up our interactive WebAPP:

  • Download all checkpoints from checkpoints and put them into a ./checkpoint folder:

    • ./checkpoint/stylegan_pretrain: Download the pre-trained checkpoint from StyleGAN2 and convert the tensorflow checkpoint to pytorch. We also released the converted checkpoint for your convenience.
    • ./checkpoint/encoder_pretrain: Pre-trained encoder.
    • ./checkpoint/encoder_pretrain/testing_embedding: Test image embeddings.
    • ./checkpoint/encoder_pretrain/training_embedding: Training image embeddings.
    • ./checkpoint/datasetgan_pretrain: Pre-trained DatasetGAN (segmentation branch).
  • Run the app using python run_app.py.

  • The app is then deployed on the web browser at locolhost:8888.

Training your own model

Here, we provide step-by-step instructions to create a new EditGAN model. We use our fully released car class as an example.

  • Step 0: Train StyleGAN.

    • Download StyleGAN training images from LSUN.

    • Train your own StyleGAN model using the official StyleGAN2 code and convert the tensorflow checkpoint to pytorch. Note the specific "stylegan_checkpoint" fields in experiments/datasetgan_car.json ; experiments/encoder_car.json ; experiments/tool_car.json.

  • Step 1: Train StyleGAN Encoder.

    • Specify location of StyleGAN checkpoint in the "stylegan_checkpoint" field in experiments/encoder_car.json.

    • Specify path with training images downloaded in Step 0 in the "training_data_path" field in experiments/encoder_car.json.

    • Run python train_encoder.py --exp experiments/encoder_car.json.

  • Step 2: Train DatasetGAN.

    • Specify "stylegan_checkpoint" field in experiments/datasetgan_car.json.

    • Download DatasetGAN training images and annotations from drive and fill in "annotation_mask_path" in experiments/datasetgan_car.json.

    • Embed DatasetGAN training images in latent space using

      python train_encoder.py --exp experiments/encoder_car.json --resume *encoder checkppoint* --testing_path data/annotation_car_32_clean --latent_sv_folder model_encoder/car_batch_8_loss_sampling_train_stylegan2/training_embedding --test True
      

      and complete "optimized_latent_path" in experiments/datasetgan_car.json.

    • Train DatasetGAN (interpreter branch for segmentation) via

      python train_interpreter.py --exp experiments/datasetgan_car.json
      
  • Step 3: Run the app.

    • Download DatasetGAN test images and annotations from drive.

    • Embed DatasetGAN test images in latent space via

      python train_encoder.py --exp experiments/encoder_car.json --resume *encoder checkppoint* --testing_path *testing image path* --latent_sv_folder model_encoder/car_batch_8_loss_sampling_train_stylegan2/training_embedding --test True
      
    • Specify the "stylegan_checkpoint", "encoder_checkpoint", "classfier_checkpoint", "datasetgan_testimage_embedding_path" fields in experiments/tool_car.json.

    • Run the app via python run_app.py.

Citations

Please use the following citation if you use our data or code:

@inproceedings{ling2021editgan,
  title = {EditGAN: High-Precision Semantic Image Editing}, 
  author = {Huan Ling and Karsten Kreis and Daiqing Li and Seung Wook Kim and Antonio Torralba and Sanja Fidler},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year = {2021}
}

License

Copyright © 2022, NVIDIA Corporation. All rights reserved.

This work is made available under the Nvidia Source Code License-NC. Please see our main LICENSE file.

License Dependencies

For any code dependencies related to StyleGAN2, the license is the Nvidia Source Code License-NC by NVIDIA Corporation, see StyleGAN2 LICENSE.

For any code dependencies related to DatasetGAN, the license is the MIT License, see DatasetGAN LICENSE.

The dataset of DatasetGAN is released under the Creative Commons BY-NC 4.0 license by NVIDIA Corporation.

For any code dependencies related to the frontend tool (including html, css and Javascript), the license is the Nvidia Source Code License-NC. To view a copy of this license, visit ./static/LICENSE.md. To view a copy of terms of usage, visit ./static/term.txt.

This code uses generative adversarial networks to generate diverse task allocation plans for Multi-agent teams.

Mutli-agent task allocation This code uses generative adversarial networks to generate diverse task allocation plans for Multi-agent teams. To change

Biorobotics Lab 5 Oct 12, 2022
The 2nd place solution of 2021 google landmark retrieval on kaggle.

Google_Landmark_Retrieval_2021_2nd_Place_Solution The 2nd place solution of 2021 google landmark retrieval on kaggle. Environment We use cuda 11.1/pyt

229 Dec 13, 2022
A MNIST-like fashion product database. Benchmark

Fashion-MNIST Table of Contents Why we made Fashion-MNIST Get the Data Usage Benchmark Visualization Contributing Contact Citing Fashion-MNIST License

Zalando Research 10.5k Jan 08, 2023
Seeing All the Angles: Learning Multiview Manipulation Policies for Contact-Rich Tasks from Demonstrations

Seeing All the Angles: Learning Multiview Manipulation Policies for Contact-Rich Tasks from Demonstrations Trevor Ablett, Daniel (Yifan) Zhai, Jonatha

STARS Laboratory 3 Feb 01, 2022
Hybrid CenterNet - Hybrid-supervised object detection / Weakly semi-supervised object detection

Hybrid-Supervised Object Detection System Object detection system trained by hybrid-supervision/weakly semi-supervision (HSOD/WSSOD): This project is

5 Dec 10, 2022
The project is an official implementation of our CVPR2019 paper "Deep High-Resolution Representation Learning for Human Pose Estimation"

Deep High-Resolution Representation Learning for Human Pose Estimation (CVPR 2019) News [2020/07/05] A very nice blog from Towards Data Science introd

Leo Xiao 3.9k Jan 05, 2023
Code for DeepCurrents: Learning Implicit Representations of Shapes with Boundaries

DeepCurrents | Webpage | Paper DeepCurrents: Learning Implicit Representations of Shapes with Boundaries David Palmer*, Dmitriy Smirnov*, Stephanie Wa

Dima Smirnov 36 Dec 08, 2022
Official Pytorch implementation for video neural representation (NeRV)

NeRV: Neural Representations for Videos (NeurIPS 2021) Project Page | Paper | UVG Data Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser-Nam Lim, Abhinav S

hao 214 Dec 28, 2022
This repository is dedicated to developing and maintaining code for experiments with wide neural networks.

Wide-Networks This repository contains the code of various experiments on wide neural networks. In particular, we implement classes for abc-parameteri

Karl Hajjar 0 Nov 02, 2021
FairMOT - A simple baseline for one-shot multi-object tracking

FairMOT - A simple baseline for one-shot multi-object tracking

Yifu Zhang 3.6k Jan 08, 2023
Dataset Condensation with Contrastive Signals

Dataset Condensation with Contrastive Signals This repository is the official implementation of Dataset Condensation with Contrastive Signals (DCC). T

3 May 19, 2022
Pytorch implementation for Semantic Segmentation/Scene Parsing on MIT ADE20K dataset

Semantic Segmentation on MIT ADE20K dataset in PyTorch This is a PyTorch implementation of semantic segmentation models on MIT ADE20K scene parsing da

MIT CSAIL Computer Vision 4.5k Jan 08, 2023
Implementing a simplified copy of Shazam application from scratch using MinHashing and LSH.

Building Shazam from scratch In this repository we tried to implement a simplified copy of the Shazam application able to tell you the name of a song

Arturo Ghinassi 0 Nov 17, 2022
Reproduces ResNet-V3 with pytorch

ResNeXt.pytorch Reproduces ResNet-V3 (Aggregated Residual Transformations for Deep Neural Networks) with pytorch. Tried on pytorch 1.6 Trains on Cifar

Pau Rodriguez 481 Dec 23, 2022
Measuring if attention is explanation with ROAR

NLP ROAR Interpretability Official code for: Evaluating the Faithfulness of Importance Measures in NLP by Recursively Masking Allegedly Important Toke

Andreas Madsen 19 Nov 13, 2022
A public available dataset for road boundary detection in aerial images

Topo-boundary This is the official github repo of paper Topo-boundary: A Benchmark Dataset on Topological Road-boundary Detection Using Aerial Images

Zhenhua Xu 79 Jan 04, 2023
Noether Networks: meta-learning useful conserved quantities

Noether Networks: meta-learning useful conserved quantities This repository contains the code necessary to reproduce experiments from "Noether Network

Dylan Doblar 33 Nov 23, 2022
This repository implements variational graph auto encoder by Thomas Kipf.

Variational Graph Auto-encoder in Pytorch This repository implements variational graph auto-encoder by Thomas Kipf. For details of the model, refer to

DaehanKim 215 Jan 02, 2023
PyTorch code for EMNLP 2021 paper: Don't be Contradicted with Anything! CI-ToD: Towards Benchmarking Consistency for Task-oriented Dialogue System

PyTorch code for EMNLP 2021 paper: Don't be Contradicted with Anything! CI-ToD: Towards Benchmarking Consistency for Task-oriented Dialogue System

Libo Qin 25 Sep 06, 2022
JUSTICE: A Benchmark Dataset for Supreme Court’s Judgment Prediction

JUSTICE: A Benchmark Dataset for Supreme Court’s Judgment Prediction CSCI 544 Final Project done by: Mohammed Alsayed, Shaayan Syed, Mohammad Alali, S

Smit Patel 3 Dec 28, 2022