[CVPR 2021] Pytorch implementation of Hijack-GAN: Unintended-Use of Pretrained, Black-Box GANs

Last update: Sep 05, 2022

Overview

Hijack-GAN: Unintended-Use of Pretrained, Black-Box GANs

In this work, we propose a framework HijackGAN, which enables non-linear latent space traversal and gain high-level controls, e.g., attributes, head poses, and landmarks, over unconditional image generation GANs in a fully black-box setting. It opens up the possibility of reusing GANs while raising concerns about unintended usage.

[Paper (CVPR 2021)][Project Page]

Prerequisites

Install required packages

pip install -r requirements.txt

Download pretrained GANs

Download the CelebAHQ pretrained weights of ProgressiveGAN [paper][code] and StyleGAN [paper][code], and then put those weights in ./models/pretrain. For example,

pretrain/
├── Pretrained_Models_Should_Be_Placed_Here
├── karras2018iclr-celebahq-1024x1024.pkl
├── karras2019stylegan-celebahq-1024x1024.pkl
├── pggan_celebahq_z.pt
├── stylegan_celebahq_z.pt
├── stylegan_headpose_z_dp.pt
└── stylegan_landmark_z.pt

Quick Start

Specify number of images to edit, a model to generate images, some parameters for editting.

LATENT_CODE_NUM=1
python edit.py \
    -m pggan_celebahq \
    -b boundaries/ \
    -n "$LATENT_CODE_NUM" \
    -o results/stylegan_celebahq_eyeglasses \
    --step_size 0.2 \
    --steps 40 \
    --attr_index 0 \
    --task attribute \
    --method ours

Usage

Important: For different given images (initial points), different step size and steps may be considered. In the following examples, we provide the parameters used in our paper. One could adjust them for better performance.

Specify Number of Samples

LATENT_CODE_NUM=1

Unconditional Modification

python edit.py \
    -m pggan_celebahq \
    -b boundaries/ \
    -n "$LATENT_CODE_NUM" \
    -o results/stylegan_celebahq_smile_editing \
    --step_size 0.2 \
    --steps 40 \
    --attr_index 0\
    --task attribute

Conditional Modification

python edit.py \
    -m pggan_celebahq \
    -b boundaries/ \
    -n "$LATENT_CODE_NUM" \
    -o results/stylegan_celebahq_smile_editing \
    --step_size 0.2 \
    --steps 40 \
    --attr_index 0\
    --condition\
    -i codes/pggan_cond/age.npy
    --task attribute

Head pose

Pitch

python edit.py \
    -m stylegan_celebahq \
    -b boundaries/ \
    -n "$LATENT_CODE_NUM" \
    -o results/ \
    --task head_pose \
    --method ours \
    --step_size 0.01 \
    --steps 2000 \
    --attr_index 1\
    --condition\
    --direction -1 \
    --demo

Yaw

python edit.py \
    -m stylegan_celebahq \
    -b boundaries/ \
    -n "$LATENT_CODE_NUM" \
    -o results/ \
    --task head_pose \
    --method ours \
    --step_size 0.1 \
    --steps 200 \
    --attr_index 0\
    --condition\
    --direction 1\
    --demo

Landmarks

Parameters for reference: (attr_index, step_size, steps) (4: 0.005 400) (5: 0.01 100), (6: 0.1 200), (8 0.1 200)

CUDA_VISIBLE_DEVICES=0 python edit.py \
    -m stylegan_celebahq \
    -b boundaries/ \
    -n "$LATENT_CODE_NUM" \
    -o results/ \
    --task landmark \
    --method ours \
    --step_size 0.1 \
    --steps 200 \
    --attr_index 6\
    --condition\
    --direction 1 \
    --demo

Generate Balanced Data

This a templeate showing how we generated balanced data for attribute manipulation (16 attributes in our internal experiments). You can modify it to fit your task better. Please first refer to here and replace YOUR_TASK_MODEL with your own classification model, and then run:

NUM=500000
CUDA_VISIBLE_DEVICES=0 python generate_balanced_data.py -m stylegan_celebahq \
    -o ./generated_data -K ./generated_data/indices.pkl -n "$NUM" -SI 0 --no_generated_imgs

[CVPR 2021] Pytorch implementation of Hijack-GAN: Unintended-Use of Pretrained, Black-Box GANs

Related tags

Overview

Hijack-GAN: Unintended-Use of Pretrained, Black-Box GANs

Prerequisites

Install required packages

Download pretrained GANs

Quick Start

Usage

Specify Number of Samples

Unconditional Modification

Conditional Modification

Head pose

Pitch

Yaw

Landmarks

Generate Balanced Data

Evaluations

TO-DO

Acknowledgment

Owner

Hui-Po Wang

Complete-IoU (CIoU) Loss and Cluster-NMS for Object Detection and Instance Segmentation (YOLACT)

🔊 Audio and fastai v2

This repository is for Contrastive Embedding Distribution Refinement and Entropy-Aware Attention Network (CEDR)

Efficient and intelligent interactive segmentation annotation software

Deeper insights into graph convolutional networks for semi-supervised learning

We present a regularized self-labeling approach to improve the generalization and robustness properties of fine-tuning.

Deep GPs built on top of TensorFlow/Keras and GPflow

Code for EMNLP2021 paper "Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training"

We utilize deep reinforcement learning to obtain favorable trajectories for visual-inertial system calibration.

An Implementation of SiameseRPN with Feature Pyramid Networks

LibFewShot: A Comprehensive Library for Few-shot Learning.

To prepare an image processing model to classify the type of disaster based on the image dataset

Ipython notebook presentations for getting starting with basic programming, statistics and machine learning techniques

SatelliteNeRF - PyTorch-based Neural Radiance Fields adapted to satellite domain

Riemannian Convex Potential Maps

Text completion with Hugging Face and TensorFlow.js running on Node.js

SSD-based Object Detection in PyTorch

A fast, distributed, high performance gradient boosting (GBT, GBDT, GBRT, GBM or MART) framework based on decision tree algorithms, used for ranking, classification and many other machine learning tasks.

Styled Augmented Translation

Raster Vision is an open source Python framework for building computer vision models on satellite, aerial, and other large imagery sets