Unofficial PyTorch Implementation for HifiFace (https://arxiv.org/abs/2106.09965)

Overview

HifiFace — Unofficial Pytorch Implementation

Image source: HifiFace: 3D Shape and Semantic Prior Guided High Fidelity Face Swapping (figure 1, pg. 1)

issueBadge starBadge repoSize lastCommit

This repository is an unofficial implementation of the face swapping model proposed by Wang et. al in their paper HifiFace: 3D Shape and Semantic Prior Guided High Fidelity Face Swapping. This implementation makes use of the Pytorch Lighting library, a light-weight wrapper for PyTorch.

HifiFace Overview

The task of face swapping applies the face and the identity of the source person to the head of the target.

The HifiFace architecture can be broken up into three primary structures. The 3D shape-aware identity extractor, the semantic facial fusion module, and an encoder-decoder structure. A high-level overview of the architecture can be seen in the image below.

Image source: HifiFace: 3D Shape and Semantic Prior Guided High Fidelity Face Swapping (figure 2, pg. 3)

Changes from the original paper

Dataset

In the paper, the author used VGGFace2 and Asian-Celeb as the training dataset. Unfortunately, the Asian-Celeb dataset can only be accessed with a Baidu account, which we do not have. Thus, we only use VGGFace2 for our training dateset.

Model

The paper proposes two versions of HifiFace model based on the output image size: 256x256 and 512x512 (referred to as Ours-256 and Ours-512 in the paper). The 512x512 model uses an extra data preprocessing before training. In this open source project, we implement the 256x256 model. For the discriminator, the original paperuses the discriminator from StarGAN v2. Our implementation uses the multi-scale discriminator from SPADE.

Installation

Build Docker Image

git clone https://github.com/mindslab-ai/hififace 
cd hififace
git clone https://github.com/sicxu/Deep3DFaceRecon_pytorch && git clone https://github.com/NVlabs/nvdiffrast && git clone https://github.com/deepinsight/insightface.git
cp -r insightface/recognition/arcface_torch/ Deep3DFaceRecon_pytorch/models/
cp -r insightface/recognition/arcface_torch/ ./model/
rm -rf insightface
cp -rf 3DMM/* Deep3DFaceRecon_pytorch
mv Deep3DFaceRecon_pytorch model/
rm -rf 3DMM
docker build -t hififace:latent .
rm -rf nvdiffrast

This Dockerfile was inspired by @yuzhou164, this issue from Deep3DFaceRecon_pytorch.

Pre-Trained Model for Deep3DFace PyTorch

Follow the guideline in Prepare prerequisite models

Set up at ./mode/Deep3DFaceRecon_pytorch/

Pre-Trained Models for ArcFace

We used official Arcface per-trained pytorch implementation Download pre-trained checkpoint from onedrive (IResNet-100 trained on MS1MV3)

Download HifiFace Pre-Trained Model

google drive link trained on VGGFace2, 300K iterations

Training

Dataset & Preprocessing

Align & Crop

We aligned the face images with the landmark extracted by 3DDFA_V2. The code will be added.

Face Segmentation Map

After finishing aligning the face images, you need to get the face segmentation map for each face images. We used face segmentation model that PSFRGAN provides. You can use their code and pre-trained model.

Dataset Folder Structure

Each face image and the corresponding segmentation map should have the same name and the same relative path from the top-level directory.

face_image_dataset_folder
└───identity1
│   │   image1.png
│   │   image2.png
│   │   ...
│   
└───identity2
│   │   image1.png
│   │   image2.png
│   │   ...
│ 
|   ...

face_segmentation_mask_folder
└───identity1
│   │   image1.png
│   │   image2.png
│   │   ...
│   
└───identity2
│   │   image1.png
│   │   image2.png
│   │   ...
│ 
|   ...

Wandb

Wandb is a powerful tool to manage your model training. Please make a wandb account and a wandb project for training HifiFace with our training code.

Changing the Configuration

  • config/model.yaml

    • dataset.train.params.image_root: directory path to the training dataset images
    • dataset.train.params.parsing_root: directory path to the training dataset parsing images
    • dataset.validation.params.image_root: directory path to the validation dataset images
    • dataset.validation.params.parsing_root: directory path to the validation dataset parsing images
  • config/trainer.yaml

    • checkpoint.save_dir: directory where the checkpoints will be saved
    • wandb: fill out your wandb entity and project name

Run Docker Container

docker run -it --ipc host --gpus all -v /PATH_TO/hififace:/workspace -v /PATH_TO/DATASET/FOLDER:/DATA --name hififace hififace:latent

Run Training Code

python hififace_trainer.py --model_config config/model.yaml --train_config config/trainer.yaml -n hififace

Inference

Single Image

python hififace_inference --gpus 0 --model_config config/model.yaml --model_checkpoint_path hififace_opensouce_299999.ckpt --source_image_path asset/inference_sample/01_source.png --target_image_path asset/inference_sample/01_target.png --output_image_path ./01_result.png

All Posible Pairs of Images in Directory

python hififace_inference --gpus 0 --model_config config/model.yaml --model_checkpoint_path hififace_opensouce_299999.ckpt  --input_directory_path asset/inference_sample --output_image_path ./result.png

Interpolation

# interpolates both the identity and the 3D shape.
python hififace_inference --gpus 0 --model_config config/model.yaml --model_checkpoint_path hififace_opensouce_299999.ckpt --source_image_path asset/inference_sample/01_source.png --target_image_path asset/inference_sample/01_target.png --output_image_path ./01_result_all.gif  --interpolation_all 

# interpolates only the identity.
python hififace_inference --gpus 0 --model_config config/model.yaml --model_checkpoint_path hififace_opensouce_299999.ckpt --source_image_path asset/inference_sample/01_source.png --target_image_path asset/inference_sample/01_target.png --output_image_path ./01_result_identity.gif  --interpolation_identity

# interpolates only the 3D shape.
python hififace_inference --gpus 0 --model_config config/model.yaml --model_checkpoint_path hififace_opensouce_299999.ckpt --source_image_path asset/inference_sample/01_source.png --target_image_path asset/inference_sample/01_target.png --output_image_path ./01_result_3d.gif  --interpolation_3d

Our Results

The results from our pre-trained model.

GIF interpolaiton results from Obama to Trump to Biden back to Obama. The left image interpolates both the identity and the 3D shape. The middle image interpolates only the identity. The right image interpolates only the 3D shape.

To-Do List

  • Pre-processing Code
  • Colab Notebook

License

BSD 3-Clause License.

Implementation Author

Changho Choi @ MINDs Lab, Inc. ([email protected])

Matthew B. Webster @ MINDs Lab, Inc. ([email protected])

Citations

@article{DBLP:journals/corr/abs-2106-09965,
  author    = {Yuhan Wang and
               Xu Chen and
               Junwei Zhu and
               Wenqing Chu and
               Ying Tai and
               Chengjie Wang and
               Jilin Li and
               Yongjian Wu and
               Feiyue Huang and
               Rongrong Ji},
  title     = {HifiFace: 3D Shape and Semantic Prior Guided High Fidelity Face Swapping},
  journal   = {CoRR},
  volume    = {abs/2106.09965},
  year      = {2021}
}
Owner
MINDs Lab
MINDsLab provides AI platform and various AI engines based on deep machine learning.
MINDs Lab
Capsule endoscopy detection DACON challenge

capsule_endoscopy_detection (DACON Challenge) Overview Yolov5, Yolor, mmdetection기반의 모델을 사용 (총 11개 모델 앙상블) 모든 모델은 학습 시 Pretrained Weight을 yolov5, yolo

MAILAB 11 Nov 25, 2022
Taming Transformers for High-Resolution Image Synthesis

Taming Transformers for High-Resolution Image Synthesis CVPR 2021 (Oral) Taming Transformers for High-Resolution Image Synthesis Patrick Esser*, Robin

CompVis Heidelberg 3.5k Jan 03, 2023
A script depending on VASP output for calculating Fermi-Softness.

Fermi softness calculation for Vienna Ab initio Simulation Package (VASP) Update 1.1.0: Big update: Rewrote the code. Use Bader atomic division instea

qslin 11 Nov 08, 2022
Reviatalizing Optimization for 3D Human Pose and Shape Estimation: A Sparse Constrained Formulation

Reviatalizing Optimization for 3D Human Pose and Shape Estimation: A Sparse Constrained Formulation This is the implementation of the approach describ

Taosha Fan 47 Nov 15, 2022
StyleSwin: Transformer-based GAN for High-resolution Image Generation

StyleSwin This repo is the official implementation of "StyleSwin: Transformer-based GAN for High-resolution Image Generation". By Bowen Zhang, Shuyang

Microsoft 349 Dec 28, 2022
TensorFlow 101: Introduction to Deep Learning for Python Within TensorFlow

TensorFlow 101: Introduction to Deep Learning I have worked all my life in Machine Learning, and I've never seen one algorithm knock over its benchmar

Sefik Ilkin Serengil 896 Jan 04, 2023
[CVPR 2020] Transform and Tell: Entity-Aware News Image Captioning

Transform and Tell: Entity-Aware News Image Captioning This repository contains the code to reproduce the results in our CVPR 2020 paper Transform and

Alasdair Tran 85 Dec 13, 2022
Pytorch implementation of Deep Recursive Residual Network for Super Resolution (DRRN)

DRRN-pytorch This is an unofficial implementation of "Deep Recursive Residual Network for Super Resolution (DRRN)", CVPR 2017 in Pytorch. [Paper] You

yun_yang 192 Dec 12, 2022
Canonical Appearance Transformations

CAT-Net: Learning Canonical Appearance Transformations Code to accompany our paper "How to Train a CAT: Learning Canonical Appearance Transformations

STARS Laboratory 54 Dec 24, 2022
NIMA: Neural IMage Assessment

PyTorch NIMA: Neural IMage Assessment PyTorch implementation of Neural IMage Assessment by Hossein Talebi and Peyman Milanfar. You can learn more from

Kyryl Truskovskyi 293 Dec 30, 2022
A tf.keras implementation of Facebook AI's MadGrad optimization algorithm

MADGRAD Optimization Algorithm For Tensorflow This package implements the MadGrad Algorithm proposed in Adaptivity without Compromise: A Momentumized,

20 Aug 18, 2022
Official pytorch implementation of the AAAI 2021 paper Semantic Grouping Network for Video Captioning

Semantic Grouping Network for Video Captioning Hobin Ryu, Sunghun Kang, Haeyong Kang, and Chang D. Yoo. AAAI 2021. [arxiv] Environment Ubuntu 16.04 CU

Hobin Ryu 43 Nov 25, 2022
This is the face keypoint train code of project face-detection-project

face-key-point-pytorch 1. Data structure The structure of landmarks_jpg is like below: |--landmarks_jpg |----AFW |------AFW_134212_1_0.jpg |------AFW_

I‘m X 3 Nov 27, 2022
Training DALL-E with volunteers from all over the Internet using hivemind and dalle-pytorch (NeurIPS 2021 demo)

Training DALL-E with volunteers from all over the Internet This repository is a part of the NeurIPS 2021 demonstration "Training Transformers Together

<a href=[email protected]"> 19 Dec 13, 2022
Repo for CVPR2021 paper "QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information"

QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information by Masato Tamura, Hiroki Ohashi, and Tomoaki Yosh

105 Dec 23, 2022
Drone Task1 - Drone Task1 With Python

Drone_Task1 Matching Results 3.mp4 1.mp4

MLV Lab (Machine Learning and Vision Lab at Korea University) 11 Nov 14, 2022
Code repository for Self-supervised Structure-sensitive Learning, CVPR'17

Self-supervised Structure-sensitive Learning (SSL) Ke Gong, Xiaodan Liang, Xiaohui Shen, Liang Lin, "Look into Person: Self-supervised Structure-sensi

Clay Gong 219 Dec 29, 2022
Codes for building and training the neural network model described in Domain-informed neural networks for interaction localization within astroparticle experiments.

Domain-informed Neural Networks Codes for building and training the neural network model described in Domain-informed neural networks for interaction

DIDACTS 0 Dec 13, 2021
Video Frame Interpolation with Transformer (CVPR2022)

VFIformer Official PyTorch implementation of our CVPR2022 paper Video Frame Interpolation with Transformer Dependencies python = 3.8 pytorch = 1.8.0

DV Lab 63 Dec 16, 2022
AI-Bot - 一个基于watermelon改造的OpenAI-GPT-2的智能机器人

AI-Bot 一个基于watermelon改造的OpenAI-GPT-2的智能机器人 在Binder上直接运行测试 目前有两种实现方式 TF2的GPT-2 TF

9 Nov 16, 2022