A collection of resources on GAN Inversion.

Overview

awesome gan-inversion papers

Awesome Maintenance PR's Welcome

This repo is a collection of resources on GAN inversion, as a supplement for our survey:

@article{xia2021survey,
  author    = {Xia, Weihao and Zhang, Yulun and Yang, Yujiu and Xue, Jing-Hao and Zhou, Bolei and Yang, Ming-Hsuan},
  title     = {GAN Inversion: A Survey},
  journal={arXiv preprint arXiv: 2101.05278},
  year={2021}
}

Contributing

Feedback and contributions are welcome!

If you think I have missed out on something (or) have any suggestions (papers, implementations and other resources), feel free to pull a request.

I have released the latex files. Please pull a request, open an issue, or send me an email if you find any inappropriate expressions of the survey.

markdown format:

**Here is the Paper Name.**
*[Author 1](homepage), Author 2, and Author 3.*
Conference or Journal Year. [[PDF](link)] [[Project](link)] [[Github](link)] [[Video](link)] [[Data](link)]

Survey

[Papers on Generative Modeling]

GAN Inversion: A Survey.
Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, Ming-Hsuan Yang.
arxiv 2021. [PDF]

inverted pretrained model

StyleGAN2-Ada: Training Generative Adversarial Networks with Limited Data.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, Timo Aila.
NeurIPS 2020. [PDF] [Github] [Steam StyleGAN2-ADA]

StyleGAN2: Analyzing and Improving the Image Quality of StyleGAN.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, Timo Aila.
CVPR 2020. [PDF] [Offical TF] [PyTorch] [Unoffical Tensorflow 2.0]

StyleGAN: A Style-Based Generator Architecture for Generative Adversarial Networks.
Tero Karras, Samuli Laine, Timo Aila.
CVPR 2019. [PDF] [Offical TF]

ProGAN: Progressive Growing of GANs for Improved Quality, Stability, and Variation.
Tero Karras, Timo Aila, Samuli Laine, Jaakko Lehtinen.
ICLR 2018. [PDF] [Offical TF]

inversion method

This part contatins generatal inversion methods, while methods in the next application part are mainly designed for specific tasks.

Using Latent Space Regression to Analyze and Leverage Compositionality in GANs.
Lucy Chai, Jonas Wulff, Phillip Isola.
ICLR 2021. [PDF] [Github] [Project] [Colab]

Hijack-GAN: Unintended-Use of Pretrained, Black-Box GANs.
Hui-Po Wang, Ning Yu, Mario Fritz.
CVPR 2021. [PDF]

e4e: Designing an Encoder for StyleGAN Image Manipulation.
Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, Daniel Cohen-Or.
arxiv 2021. [PDF] [Github]

Enjoy Your Editing: Controllable GANs for Image Editing via Latent Space Navigation.
Peiye Zhuang, Oluwasanmi Koyejo, Alexander G. Schwing.
ICLR 2021. [PDF]

Improved StyleGAN Embedding: Where are the Good Latents?
Peihao Zhu, Rameen Abdal, Yipeng Qin, Peter Wonka.
arxiv 2020. [PDF]

Learning a Deep Reinforcement Learning Policy Over the Latent Space of a Pre-trained GAN for Semantic Age Manipulation.
Kumar Shubham, Gopalakrishnan Venkatesh, Reijul Sachdev, Akshi, Dinesh Babu Jayagopi, G. Srinivasaraghavan.
arxiv 2020. [PDF]

Lifting 2D StyleGAN for 3D-Aware Face Generation.
Yichun Shi, Divyansh Aggarwal, Anil K. Jain.
arxiv 2020. [PDF]

Navigating the GAN Parameter Space for Semantic Image Editing.
Anton Cherepkov, Andrey Voynov, Artem Babenko.
arxiv 2020. [PDF] [Github]

Augmentation-Interpolative AutoEncoders for Unsupervised Few-Shot Image Generation.
Davis Wertheimer, Omid Poursaeed, Bharath Hariharan.
arxiv 2020. [PDF]

Mask-Guided Discovery of Semantic Manifolds in Generative Models.
Mengyu Yang, David Rokeby, Xavier Snelgrove.
Workshop on Machine Learning for Creativity and Design (NeurIPS) 2020. [PDF] [Github]

Unsupervised Discovery of Disentangled Manifolds in GANs.
Yu-Ding Lu, Hsin-Ying Lee, Hung-Yu Tseng, Ming-Hsuan Yang.
arxiv 2020. [PDF]]

StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation.
Zongze Wu, Dani Lischinski, Eli Shechtman.
arxiv 2020. [PDF]

GAN Steerability without optimization.
Nurit Spingarn-Eliezer, Ron Banner, Tomer Michaeli.
ICLR 2021. [OpenReview] [PDF]

On The Inversion Of Deep Generative Models (When and How Can Deep Generative Models be Inverted?).
Aviad Aberdam, Dror Simon, Michael Elad.
arxiv 2020. [PDF] [OpenReview]

PIE: Portrait Image Embedding for Semantic Control.
A. Tewari, M. Elgharib, M. BR, F. Bernard, H-P. Seidel, P. P‌érez, M. Zollhöfer, C.Theobalt.
SIGGRAPH Asia 2020. [PDF] [Project]

Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation.
Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, Daniel Cohen-Or.
CVPR 2021. [PDF] [Github] [Project]

GAN-Control: Explicitly Controllable GANs.
Alon Shoshan, Nadav Bhonker, Igor Kviatkovsky, Gerard Medioni.
arxiv 2021. [PDF]

Understanding the Role of Individual Units in a Deep Neural Network.
David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, Antonio Torralba.
National Academy of Sciences 2020. [PDF] [Github] [Project]

GHFeat: Generative Hierarchical Features from Synthesizing Images.
Yinghao Xu, Yujun Shen, Jiapeng Zhu, Ceyuan Yang, Bolei Zhou.
CVPR 2021. [PDF] [Github] [Project]

SeFa: Closed-Form Factorization of Latent Semantics in GANs.
Yujun Shen, Bolei Zhou.
CVPR 2021. [PDF] [Github] [Project]

Collaborative Learning for Faster StyleGAN Embedding.
Shanyan Guan, Ying Tai, Bingbing Ni, Feida Zhu, Feiyue Huang, Xiaokang Yang.
arxiv 2020. [PDF]

Disentangling in Latent Space by Harnessing a Pretrained Generator.
Yotam Nitzan, Amit Bermano, Yangyan Li, Daniel Cohen-Or.
arxiv 2020. [PDF]

Face Identity Disentanglement via Latent Space Mapping.
Yotam Nitzan, Amit Bermano, Yangyan Li, Daniel Cohen-Or.
SIGGRAPH Asia (TOG) 2020. [PDF] [Github]

Transforming and Projecting Images into Class-conditional Generative Networks.
Minyoung Huh, Richard Zhang, Jun-Yan Zhu, Sylvain Paris, Aaron Hertzmann.
ECCV 2020. [PDF] [Github] [Project]

Interpreting the Latent Space of GANs via Correlation Analysis for Controllable Concept Manipulation.
Ziqiang Li, Rentuo Tao, Hongjing Niu, Bin Li.
arxiv 2020. [PDF]

Improving Inversion and Generation Diversity in StyleGAN using a Gaussianized Latent Space.
Jonas Wulff, Antonio Torralba.
arxiv 2020. [PDF]

GANSpace: Discovering Interpretable GAN Controls.
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, Sylvain Paris.
NeurIPS 2020. [PDF] [Github]

MimicGAN: Robust Projection onto Image Manifolds with Corruption Mimicking.
Rushil Anirudh, Jayaraman J. Thiagarajan, Bhavya Kailkhura, Timo Bremer.
IJCV 2020. [PDF]

StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows.
Rameen Abdal, Peihao Zhu, Niloy Mitra, Peter Wonka.
Siggraph (TOG) 2021. [PDF] [Github]

Rewriting a Deep Generative Model.
David Bau, Steven Liu, Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba.
ECCV 2020. [PDF] [Github]

StyleGAN2 Distillation for Feed-forward Image Manipulation.
Yuri Viazovetskyi, Vladimir Ivashkin, Evgeny Kashin.
ECCV 2020. [PDF] [Github]

In-Domain GAN Inversion for Real Image Editing.
Jiapeng Zhu, Yujun Shen, Deli Zhao, Bolei Zhou.
ECCV 2020. [PDF] [Project] [Github]

Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation.
Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin, Chen Change Loy, Ping Luo.
ECCV 2020. [PDF] [Github]

On the "steerability" of generative adversarial networks.
Ali Jahanian, Lucy Chai, Phillip Isola.
ICLR 2020. [PDF] [Project]

Unsupervised Discovery of Interpretable Directions in the GAN Latent Space.
Andrey Voynov, Artem Babenko.
ICML 2020. [PDF] [Github]

Your Local GAN: Designing Two Dimensional Local Attention Mechanisms for Generative Models.
Giannis Daras, Augustus Odena, Han Zhang, Alexandros G. Dimakis.
CVPR 2020. [PDF]

A Disentangling Invertible Interpretation Network for Explaining Latent Representations.
Patrick Esser, Robin Rombach, Björn Ommer.
CVPR 2020. [PDF] [Project] [Github]

Editing in Style: Uncovering the Local Semantics of GANs.
Edo Collins, Raja Bala, Bob Price, Sabine Süsstrunk.
CVPR 2020. [PDF] [Github]

Image Processing Using Multi-Code GAN Prior.
Jinjin Gu, Yujun Shen, Bolei Zhou.
CVPR 2020. [PDF] [Project] [Github]

Interpreting the Latent Space of GANs for Semantic Face Editing.
Yujun Shen, Jinjin Gu, Xiaoou Tang, Bolei Zhou.
CVPR 2020. [PDF] [Project] [Github]

Image2StyleGAN++: How to Edit the Embedded Images?
Rameen Abdal, Yipeng Qin, Peter Wonka.
CVPR 2020. [PDF]

Semantic Photo Manipulation with a Generative Image Prior.
David Bau, Hendrik Strobelt, William Peebles, Jonas, Bolei Zhou, Jun-Yan Zhu, Antonio Torralba.
SIGGRAPH 2019. [PDF]

Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?
Rameen Abdal, Yipeng Qin, Peter Wonka.
ICCV 2019. [PDF] [Github]

Seeing What a GAN Cannot Generate.
David Bau, Jun-Yan Zhu, Jonas Wulff, William Peebles, Hendrik Strobelt, Bolei Zhou, Antonio Torralba.
ICCV 2019. [PDF] [PDF]

GAN-based Projector for Faster Recovery with Convergence Guarantees in Linear Inverse Problems.
Ankit Raj, Yuqi Li, Yoram Bresler.
ICCV 2019. [PDF]

Inverting Layers of a Large Generator.
David Bau, Jun-Yan Zhu, Jonas Wulff, William Peebles, Hendrik Strobelt, Bolei Zhou, Antonio Torralba.
ICCV 2019. [PDF]

Inverting The Generator Of A Generative Adversarial Network (II).
Antonia Creswell, Anil A Bharath.
TNNLS 2018. [PDF] [Github]

Invertibility of Convolutional Generative Networks from Partial Measurements.
Fangchang Ma, Ulas Ayaz, Sertac Karaman.
NeurIPS 2018. [PDF] [Github]

Metrics for Deep Generative Models.
Nutan Chen, Alexej Klushyn, Richard Kurle, Xueyan Jiang, Justin Bayer, Patrick van der Smagt.
AISTATS 2018. [PDF]

Towards Understanding the Invertibility of Convolutional Neural Networks.
Anna C. Gilbert, Yi Zhang, Kibok Lee, Yuting Zhang, Honglak Lee.
IJCAI 2017. [PDF]

One Network to Solve Them All - Solving Linear Inverse Problems using Deep Projection Models.
J. H. Rick Chang, Chun-Liang Li, Barnabas Poczos, B. V. K. Vijaya Kumar, Aswin C. Sankaranarayanan.
ICCV 2017. [PDF]

Precise Recovery of Latent Vectors from Generative Adversarial Networks.
Zachary C. Lipton, Subarna Tripathi.
ICLR 2017 workshop. [PDF] [Github]

Inverting The Generator Of A Generative Adversarial Network.
Antonia Creswell, Anil Anthony Bharath.
NIPSW 2016. [PDF]

Generative Visual Manipulation on the Natural Image Manifold.
Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, Alexei A. Efros.
ECCV 2016. [PDF]

application

content generation

Paint by Word.
David Bau, Alex Andonian, Audrey Cui, YeonHwan Park, Ali Jahanian, Aude Oliva, Antonio Torralba.
arxiv 2021. [PDF]

Unsupervised Image Transformation Learning via Generative Adversarial Networks.
Kaiwen Zha, Yujun Shen, Bolei Zhou.
arxiv 2021. [PDF] [Project]

TediGAN: Text-Guided Diverse Image Generation and Manipulation.
Weihao Xia, Yujiu Yang, Jing-Hao Xue, Baoyuan Wu.
CVPR 2021. [PDF] [Data] [Github]

LOHO: Latent Optimization of Hairstyles via Orthogonalization.
Rohit Saha, Brendan Duke, Florian Shkurti, Graham W. Taylor, Parham Aarabi.
CVPR 2021. [PDF] [Github]

SAM: Only a Matter of Style-Age Transformation Using a Style-Based Regression Model.
Yuval Alaluf, Or Patashnik, Daniel Cohen-Or.
arxiv 2021. [PDF] [Github]

OSTeC: One-Shot Texture Completion.
Baris Gecer, Jiankang Deng, Stefanos Zafeiriou.
arxiv 2021. [PDF] [Github]

GAN2Shape: Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs.
Xingang Pan, Bo Dai, Ziwei Liu, Chen Change Loy, Ping Luo.
ICLR 2021 (oral). [PDF] [Github] [Project]

Exploring Adversarial Fake Images on Face Manifold.
Dongze Li, Wei Wang, Hongxing Fan, Jing Dong.
arxiv 2021. [PDF]

Generating Images from Caption and Vice Versa via CLIP-Guided Generative Latent Space Search.
Federico A. Galatolo, Mario G.C.A. Cimino, Gigliola Vaglini.
arxiv 2021. [PDF]

Unsupervised Image-to-Image Translation via Pre-trained StyleGAN2 Network.
Jialu Huang, Jing Liao, Sam Kwong.
arxiv 2020. [PDF]

DeepI2I: Enabling Deep Hierarchical Image-to-Image Translation by Transferring from GANs.
yaxing wang, Lu Yu, Joost van de Weijer.
NeurIPS 2020. [PDF] [Github]

DeepLandscape: Adversarial Modeling of Landscape Videos.
E. Logacheva, R. Suvorov, O. Khomenko, A. Mashikhin, and V. Lempitsky.
ECCV 2020. [PDF] [Github] [Project]

image restoration

GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution.
Kelvin C.K. Chan, Xintao Wang, Xiangyu Xu, Jinwei Gu, Chen Change Loy.
CVPR 2021. [PDF] [Project] [Github]

GFP-GAN: Towards Real-World Blind Face Restoration with Generative Facial Prior.
Xintao Wang, Yu Li, Honglun Zhang, Ying Shan.
arxiv 2021. [PDF] [Project]

image understanding

Repurposing GANs for One-shot Semantic Part Segmentation.
Nontawat Tritrong, Pitchaporn Rewatbowornwong, Supasorn Suwajanakorn.
CVPR 2021 (oral). [PDF] [Project] [Github]

compressed sensing

Generator Surgery for Compressed Sensing.
Niklas Smedemark-Margulies, Jung Yeon Park, Max Daniels, Rose Yu, Jan-Willem van de Meent, Paul Hand.
arxiv 2021. [PDF] [Github]

Task-Aware Compressed Sensing with Generative Adversarial Networks.
Maya Kabkab, Pouya Samangouei, Rama Chellappa.
AAAI 2018. [PDF]

acknowledgement

Thanks for the feedback from Jun-Yan Zhu, Andrey Voynov, and Rushil Anirudh.

Code for the Interspeech 2021 paper "AST: Audio Spectrogram Transformer".

AST: Audio Spectrogram Transformer Introduction Citing Getting Started ESC-50 Recipe Speechcommands Recipe AudioSet Recipe Pretrained Models Contact I

Yuan Gong 603 Jan 07, 2023
Ratatoskr: Worcester Tech's conference scheduling system

Ratatoskr: Worcester Tech's conference scheduling system In Norse mythology, Ratatoskr is a squirrel who runs up and down the world tree Yggdrasil to

4 Dec 22, 2022
YOLOX + ROS(1, 2) object detection package

YOLOX + ROS(1, 2) object detection package

Ar-Ray 158 Dec 21, 2022
A PyTorch implementation of SlowFast based on ICCV 2019 paper "SlowFast Networks for Video Recognition"

SlowFast A PyTorch implementation of SlowFast based on ICCV 2019 paper SlowFast Networks for Video Recognition. Requirements Anaconda PyTorch conda in

Hao Ren 8 Dec 23, 2022
Pytorch and Keras Implementations of Hyperspectral Image Classification -- Traditional to Deep Models: A Survey for Future Prospects.

The repository contains the implementations for Hyperspectral Image Classification -- Traditional to Deep Models: A Survey for Future Prospects. Model

Ankur Deria 115 Jan 06, 2023
This code is the implementation of the paper "Coherence-Based Distributed Document Representation Learning for Scientific Documents".

Introduction This code is the implementation of the paper "Coherence-Based Distributed Document Representation Learning for Scientific Documents". If

tsc 0 Jan 11, 2022
Source code for ZePHyR: Zero-shot Pose Hypothesis Rating @ ICRA 2021

ZePHyR: Zero-shot Pose Hypothesis Rating ZePHyR is a zero-shot 6D object pose estimation pipeline. The core is a learned scoring function that compare

R-Pad - Robots Perceiving and Doing 18 Aug 22, 2022
A simple Tensorflow based library for deep and/or denoising AutoEncoder.

libsdae - deep-Autoencoder & denoising autoencoder A simple Tensorflow based library for Deep autoencoder and denoising AE. Library follows sklearn st

Rajarshee Mitra 147 Nov 18, 2022
Implicit MLE: Backpropagating Through Discrete Exponential Family Distributions

torch-imle Concise and self-contained PyTorch library implementing the I-MLE gradient estimator proposed in our NeurIPS 2021 paper Implicit MLE: Backp

UCL Natural Language Processing 249 Jan 03, 2023
Automatic Idiomatic Expression Detection

IDentifier of Idiomatic Expressions via Semantic Compatibility (DISC) An Idiomatic identifier that detects the presence and span of idiomatic expressi

5 Jun 09, 2022
This repository provides the code for MedViLL(Medical Vision Language Learner).

MedViLL This repository provides the code for MedViLL(Medical Vision Language Learner). Our proposed architecture MedViLL is a single BERT-based model

SuperSuperMoon 39 Jan 05, 2023
Optimized code based on M2 for faster image captioning training

Transformer Captioning This repository contains the code for Transformer-based image captioning. Based on meshed-memory-transformer, we further optimi

lyricpoem 16 Dec 16, 2022
Official Pytorch implementation of C3-GAN

Official pytorch implemenation of C3-GAN Contrastive Fine-grained Class Clustering via Generative Adversarial Networks [Paper] Authors: Yunji Kim, Jun

NAVER AI 114 Dec 02, 2022
A clean and robust Pytorch implementation of PPO on continuous action space.

PPO-Continuous-Pytorch I found the current implementation of PPO on continuous action space is whether somewhat complicated or not stable. And this is

XinJingHao 56 Dec 16, 2022
PyTorch implementation of the method described in the paper VoiceLoop: Voice Fitting and Synthesis via a Phonological Loop.

VoiceLoop PyTorch implementation of the method described in the paper VoiceLoop: Voice Fitting and Synthesis via a Phonological Loop. VoiceLoop is a n

Meta Archive 873 Dec 15, 2022
iBOT: Image BERT Pre-Training with Online Tokenizer

Image BERT Pre-Training with iBOT Official PyTorch implementation and pretrained models for paper iBOT: Image BERT Pre-Training with Online Tokenizer.

Bytedance Inc. 435 Jan 06, 2023
Simple is not Easy: A Simple Strong Baseline for TextVQA and TextCaps[AAAI2021]

Simple is not Easy: A Simple Strong Baseline for TextVQA and TextCaps Here is the code for ssbassline model. We also provide OCR results/features/mode

ZephyrZhuQi 51 Nov 18, 2022
CLUES: Few-Shot Learning Evaluation in Natural Language Understanding

CLUES: Few-Shot Learning Evaluation in Natural Language Understanding This repo contains the data and source code for baseline models in the NeurIPS 2

Microsoft 29 Dec 29, 2022
[ICML 2021] DouZero: Mastering DouDizhu with Self-Play Deep Reinforcement Learning | 斗地主AI

[ICML 2021] DouZero: Mastering DouDizhu with Self-Play Deep Reinforcement Learning DouZero is a reinforcement learning framework for DouDizhu (斗地主), t

Kwai Inc. 3.1k Jan 04, 2023
Stitch it in Time: GAN-Based Facial Editing of Real Videos

STIT - Stitch it in Time [Project Page] Stitch it in Time: GAN-Based Facial Edit

1.1k Jan 04, 2023