Facial Image Inpainting with Semantic Control

Overview

Facial Image Inpainting with Semantic Control

In this repo, we provide a model for the controllable facial image inpainting task. This model enables users to intuitively edit their images by using parametric 3D faces.

The technology report is comming soon.

  • Image Inpainting results

  • Fine-grained Control

Quick Start

Installation

  • Clone the repository and set up a conda environment with all dependencies as follows
git clone https://github.com/RenYurui/Controllable-Face-Inpainting.git --recursive
cd Controllable-Face-Inpainting

# 1. Create a conda virtual environment.
conda create -n cfi python=3.6
source activate cfi
conda install -c pytorch pytorch=1.7.1 torchvision cudatoolkit=10.2

# 2. install pytorch3d
conda install -c fvcore -c iopath -c conda-forge fvcore iopath
conda install -c bottler nvidiacub
git clone https://github.com/facebookresearch/pytorch3d.git
cd pytorch3d && pip install -e .

# 3. Install other dependencies
pip install -r requirements.txt

Download Prerequisite Models

  • Follow Deep3DFaceRecon to prepare ./BFM folder. Download 01_MorphableModel.mat and Expression Basis Exp_Pca.bin. Put the obtained files into the ./Deep3DFaceRecon_pytorch/BFM floder. Then link the folder to the root path.
ln -s /PATH_TO_REPO_ROOT/Deep3DFaceRecon_pytorch/BFM /PATH_TO_REPO_ROOT
  • Clone the Arcface repo
cd third_part
git clone https://github.com/deepinsight/insightface.git
cp -r ./insightface/recognition/arcface_torch/ ./

The Arcface is used to extract identity features for loss computation. Download the pre-trained model from Arcface using this link. By default, the resnet50 backbone (ms1mv3_arcface_r50_fp16) is used. Put the obtained weights into ./third_part/arcface_torch/ms1mv3_arcface_r50_fp16/backbone.pth

  • Download the pretrained weights of our model from Google Driven. Save the obtained files into folder ./result.

Inference

We provide some example images. Please run the following code for inference

CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --nproc_per_node=1 --master_port 1234 demo.py \
--config ./config/facial_image_renderer_ffhq.yaml \
--name facial_image_renderer_ffhq \
--output_dir ./visi_result \
--input_dir ./examples/inputs \
--mask_dir ./examples/masks

Train the model from scratch

Dataset Preparation

  • Download dataset. We use Celeba-HQ and FFHQ for training and inference. Please download the datasets (image format) and put them under ./dataset folder.
  • Obtain 3D faces by using Deep3DFaceRecon. Follow the Deep3DFaceRecon repo to download the trained weights. And save it as: ./Deep3DFaceRecon_pytorch/checkpoints/face_recon/epoch_20.pth
# 1. Extract keypoints from the face images for cropping.
cd scripts
# extracted keypoints from celeba
python extract_kp.py \
--data_root PATH_TO_CELEBA_ROOT \
--output_dir PATH_TO_KEYPOINTS \
--dataset celeba \
--device_ids 0,1 \
--workers 6

# 2. Extract 3DMM coefficients from the face images.
cd .. #repo root
# we provide some scripts for easy of use. However, one can use the original repo to extract the coefficients.
cp scripts/inference_options.py ./Deep3DFaceRecon_pytorch/options
cp scripts/face_recon.py ./Deep3DFaceRecon_pytorch
cp scripts/facerecon_inference_model.py ./Deep3DFaceRecon_pytorch/models
cp scripts/pytorch_3d.py ./Deep3DFaceRecon_pytorch/util
ln -s /PATH_TO_REPO_ROOT/third_part/arcface_torch /PATH_TO_REPO_ROOT/Deep3DFaceRecon_pytorch/models

cd Deep3DFaceRecon_pytorch

python face_recon.py \
--input_dir PATH_TO_CELEBA_ROOT \
--keypoint_dir PATH_TO_KEYPOINTS \
--output_dir PATH_TO_3DMM_COEFFICIENT \
--inference_batch_size 100 \
--name=face_recon \
--dataset_name celeba \
--epoch=20 \
--model facerecon_inference

# 3. Save images and the coefficients into a lmdb file.
cd .. #repo root
python prepare_data.py \
--root PATH_TO_CELEBA_ROOT \
--coeff_file PATH_TO_3DMM_COEFFICIENT \
--dataset celeba \
--out PATH_TO_CELEBA_LMDB_ROOT

Train The Model

# we first train the semantic_descriptor_recommender
CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --nproc_per_node=4 --master_port 1234 train.py \
--config ./config/semantic_descriptor_recommender_celeba.yaml \
--name semantic_descriptor_recommender_celeba

# Then, we trian the facial_image_renderer for image inpainting
CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --nproc_per_node=4 --master_port 1234 train.py \
--config ./config/facial_image_renderer_celeba.yaml \
--name facial_image_renderer_celeba
Owner
Ren Yurui
Ren Yurui
Tensorflow 2 implementation of our high quality frame interpolation neural network

FILM: Frame Interpolation for Large Scene Motion Project | Paper | YouTube | Benchmark Scores Tensorflow 2 implementation of our high quality frame in

Google Research 1.6k Dec 28, 2022
Iterative Training: Finding Binary Weight Deep Neural Networks with Layer Binarization

Iterative Training: Finding Binary Weight Deep Neural Networks with Layer Binarization This repository contains the source code for the paper (link wi

Rakuten Group, Inc. 0 Nov 19, 2021
FID calculation with proper image resizing and quantization steps

clean-fid: Fixing Inconsistencies in FID Project | Paper The FID calculation involves many steps that can produce inconsistencies in the final metric.

Gaurav Parmar 606 Jan 06, 2023
Some toy examples of score matching algorithms written in PyTorch

toy_gradlogp This repo implements some toy examples of the following score matching algorithms in PyTorch: ssm-vr: sliced score matching with variance

Ending Hsiao 21 Dec 26, 2022
Caffe models in TensorFlow

Caffe to TensorFlow Convert Caffe models to TensorFlow. Usage Run convert.py to convert an existing Caffe model to TensorFlow. Make sure you're using

Saumitro Dasgupta 2.8k Dec 31, 2022
Consistency Regularization for Adversarial Robustness

Consistency Regularization for Adversarial Robustness Official PyTorch implementation of Consistency Regularization for Adversarial Robustness by Jiho

40 Dec 17, 2022
Lightwood is Legos for Machine Learning.

Lightwood is like Legos for Machine Learning. A Pytorch based framework that breaks down machine learning problems into smaller blocks that can be glu

MindsDB Inc 312 Jan 08, 2023
Code to reproduce the results in the paper "Tensor Component Analysis for Interpreting the Latent Space of GANs".

Tensor Component Analysis for Interpreting the Latent Space of GANs [ paper | project page ] Code to reproduce the results in the paper "Tensor Compon

James Oldfield 4 Jun 17, 2022
Cobalt Strike teamserver detection.

Cobalt-Strike-det Cobalt Strike teamserver detection. usage: cobaltstrike_verify.py [-l TARGETS] [-t THREADS] optional arguments: -h, --help show this

TimWhite 17 Sep 27, 2022
Interpretable-contrastive-word-mover-s-embedding

Interpretable-contrastive-word-mover-s-embedding Paper Datasets Here is a Dropbox link to the datasets used in the paper: https://www.dropbox.com/sh/n

0 Nov 02, 2021
This repository provides the code for MedViLL(Medical Vision Language Learner).

MedViLL This repository provides the code for MedViLL(Medical Vision Language Learner). Our proposed architecture MedViLL is a single BERT-based model

SuperSuperMoon 39 Jan 05, 2023
Causal estimators for use with WhyNot

WhyNot Estimators A collection of causal inference estimators implemented in Python and R to pair with the Python causal inference library whynot. For

ZYKLS 8 Apr 06, 2022
Continual learning with sketched Jacobian approximations

Continual learning with sketched Jacobian approximations This repository contains the code for reproducing figures and results in the paper ``Provable

Machine Learning and Information Processing Laboratory 1 Jun 30, 2022
Representing Long-Range Context for Graph Neural Networks with Global Attention

Graph Augmentation Graph augmentation/self-supervision/etc. Algorithms gcn gcn+virtual node gin gin+virtual node PNA GraphTrans Augmentation methods N

UC Berkeley RISE 67 Dec 30, 2022
[NeurIPS 2021] Introspective Distillation for Robust Question Answering

Introspective Distillation (IntroD) This repository is the Pytorch implementation of our paper "Introspective Distillation for Robust Question Answeri

Yulei Niu 13 Jul 26, 2022
LRBoost is a scikit-learn compatible approach to performing linear residual based stacking/boosting.

LRBoost is a sckit-learn compatible package for linear residual boosting. LRBoost combines a linear estimator and a non-linear estimator to leverage t

Andrew Patton 5 Nov 23, 2022
Populating 3D Scenes by Learning Human-Scene Interaction https://posa.is.tue.mpg.de/

Populating 3D Scenes by Learning Human-Scene Interaction [Project Page] [Paper] License Software Copyright License for non-commercial scientific resea

Mohamed Hassan 81 Nov 08, 2022
To build a regression model to predict the concrete compressive strength based on the different features in the training data.

Cement-Strength-Prediction Problem Statement To build a regression model to predict the concrete compressive strength based on the different features

Ashish Kumar 4 Jun 11, 2022
Supplementary code for the experiments described in the 2021 ISMIR submission: Leveraging Hierarchical Structures for Few Shot Musical Instrument Recognition.

Music Trees Supplementary code for the experiments described in the 2021 ISMIR submission: Leveraging Hierarchical Structures for Few Shot Musical Ins

Hugo Flores GarcĂ­a 32 Nov 22, 2022
Provide baselines and evaluation metrics of the task: traffic flow prediction

Note: This repo is adpoted from https://github.com/UNIMIBInside/Smart-Mobility-Prediction. Due to technical reasons, I did not fork their code. Introd

Zhangzhi Peng 11 Nov 02, 2022