[ECCV 2020] Reimplementation of 3DDFAv2, including face mesh, head pose, landmarks, and more.

Overview

Stable Head Pose Estimation and Landmark Regression via 3D Dense Face Reconstruction

FaceReconstructionDemo

Language grade: Python License ECCV

Reimplementation of (ECCV 2020) Towards Fast, Accurate and Stable 3D Dense Face Alignment via Tensorflow Lite framework, face mesh, head pose, landmarks, and more.

  • CPU real-time face deteciton, alignment, and reconstruction pipeline.
  • Lightweight render library, 5x faster (3ms vs 15ms) than the Sim3DR tools.
  • Camera matrix and dense/sparse landmarks prediction via a single network.
  • Generate facial parameters for robust head pose and expression estimation.

Setup

Basic Requirements

  • Python 3.6+
  • pip3 install -r requirements.txt

Render for Dense Face

  • GCC 6.0+
  • bash build_render.sh
  • (Cautious) For Windows user, please refer to this tutorial for more details.

3D Facial Landmarks

In this project, we perform dense face reconstruction by 3DMM parameters regression. The regression target is simplified as camera matrix (C, shape of 3x4), appearance parameters (S, shape of 1x40), and expression variables (E, shape of 1x10), with 62 dimensions in total.

The sparse or dense facial landmarks can be estimated by applying these parameters to a predefined 3D model, such as BFM. More specifically, the following formula describes how to generate a face through parameters:

Generate Face

where U and W are from pre-defined face model. Combine them linearly with parameters to generate sparse or dense faces. Finally, we need to integrate the posture information into the result:

With matrix

where R (shape of 3x3) and T (shape of 3x1) denote rotation and translation matrices, respectively, which are fractured from the camera matrix C.

Sparse

sparse demo

Since we have reconstructed the entire face, the 3D face alignment can be achieved by selecting the landmarks at the corresponding positions. See [TPAMI 2017] Face alignment in full pose range: A 3d total solution for more details.

Comparing with the method of first detecting 2D landmarks and then performing depth estimation, directly fitting 3DMM to solve 3D face alignment can not only obtain more accurate results in larger pose scenes, but also has obvious advantages in speed.

We provide a demonstration script that can generate 68 landmarks based on the reconstructed face. Run the following command to view the real-time 3D face alignment results:

python3 demo_video.py -m sparse -f <your-video-path>

Dense

dense demo

Currently, our method supports up to 38,365 landmarks. We draw landmarks every 6 indexes for a better illustration. Run the demonstrate script in dense mode for real-time dense facial landmark localization:

python3 demo_video.py -m dense -f <your-video-path>

Face Reconstruction

Our network is multi-task since it can directly regress 3DMM params from a single face image for reconstruction, as well as estimate the head pose via R and T prediction. During training, the predicted R can supervise the params regression branch to generate refined face mesh. Meanwhile, the landmarks calculated via the params are provided as the labeled data to the pose estimation branch for training, through the cv2.solvePnP tool.

Theoretically speaking, the head pose estimation task in our model is weakly supervised, since only a few labeled data is required to activate the training process. In detail, the loss function at the initial stage can be described as follows:

Init Pose Loss

After the initialization process, the ground truth can be replaced by the prediction results of other branches. Therefore, the loss function will be transformed to the following equation in the later stage of the training process:

Pose Loss

In general, fitting the 3D model during the training process dynamically can avoid the inaccurate head pose estimation results caused by the coarse predefined model.

Head Pose

pose demo

Traditional head pose estimation approaches, such as Appearance Template Models, Detector Arrays, and Mainfold Embedding have been extensively studied. However, methods based on deep learning improved the prediction accuracy to meet actual needs, until recent years.

Given a set of predefined 3D facial landmarks and the corresponding 2D image projections, the SolvePnP tool can be utilized to calculate the rotation matrix. However, the adopted mean 3D human face model usually introduces intrinsic error during the fitting process. Meanwhile, the additional landmarks extraction component is also kind of cumbersome.

Therefore, we designed a network branch for directly regress 6DoF parameters from the face image. The predictions include the 3DoF rotation matrix R (Pitch, Yaw, Roll), and the 3DoF translation matrix T (x, y, z), Compared with the landmark-based method, directly regression the camera matrix is more robust and stable, as well as significantly reduce the network training cost. Run the demonstrate script in pose mode to view the real-time head pose estimation results:

python3 demo_video.py -m pose -f <your-video-path>

Expression

Expression

Coarse expression estimation can be achieved by combining the predefined expressions in BFM linearly. In our model, regression the E is one of the tasks of the params prediction branch. Obviously, the accuracy of the linear combination is positively related to the dimension.

Clipping parameters can accelerate the training process, however, it can also reduce reconstruction accuracy, especially details such as eye and mouth. More specifically, E has a greater impact on face details than S when emotion is involved. Therefore, we choose 10-dimension for a tradeoff between the speed and the accuracy, the training data can be found at here for refinement.

In addition, we provide a simple facial expression rendering script. Run the following command for illustration:

python3 demo_image.py <your-image-path>

Mesh

mesh demo

According to the predefined BFM and the predicted 3DMM parameters, the dense 3D facial landmarks can be easily calculated. On this basis, through the index mapping between the morphable triangle vertices and the dense landmarks defined in BFM, the renderer can plot these geometries with depth infomation for mesh preview. Run the demonstrate script in mesh mode for real-time face reconstruction:

python3 demo_video.py -m mesh -f <your-video-path>

Benchmark

Our network can directly output the camera matrix and sparse or dense landmarks. Compared with the model in the original paper with the same backbone, the additional parameters yield via the pose regression branch does not significantly affect the inference speed, which means it can still be CPU real-time.

Scheme THREAD=1 THREAD=2 THREAD=4
Inference 7.79ms 6.88ms 5.83ms

In addition, since most of the operations are wrapped in the model, the time consumption of pre-processing and post-processing are significantly reduced. Meanwhile, the optimized lightweight renderer is 5x faster (3ms vs 15ms) than the Sim3DR tools. These measures decline the latency of the entire pipeline.

Stage Preprocess Inference Postprocess Render
Each face cost 0.23ms 7.79ms 0.39ms 3.92ms

Run the following command for speed benchmark:

python3 video_speed_benchmark.py <your-video-path>

Citation

@inproceedings{guo2020towards,
    title={Towards Fast, Accurate and Stable 3D Dense Face Alignment},
    author={Guo, Jianzhu and Zhu, Xiangyu and Yang, Yang and Yang, Fan and Lei, Zhen and Li, Stan Z},
    booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},
    year={2020}
}
Comments
  • __init__() got an unexpected keyword argument 'num_threads'

    __init__() got an unexpected keyword argument 'num_threads'

    While running "python3 demo_video.py -m mesh -f I receive the following ERROR: File "demo_video.py", line 53, in main(args) File "demo_video.py", line 16, in main fa = service.DenseFaceReconstruction("weights/dense_face.tflite") File "/home/happy/PycharmProjects/Dense-Head-Pose-Estimation/service/TFLiteFaceAlignment.py", line 11, in init num_threads=num_threads) TypeError: init() got an unexpected keyword argument 'num_threads'

    opened by berylyellow 2
  • asset/render.so not found

    asset/render.so not found

    While running "python3 demo_video.py -m mesh -f I receive the following ERROR OSError: asset/render.so: cannot open shared object file: No such file or directory I can successfully run the other three modes (sparse, dense, pose)

    opened by fisakhan 2
  • How can i get Depth

    How can i get Depth

    hi, I want to get information about the depth of the face, how do I calculate it? i don't need rendering but only need depth map. like https://github.kakaocorp.com/data-sci/3DDFA_V2_tf2/blob/master/utils/depth.py thank you in advance!

    opened by joongkyunKim 1
  • there is no asset/render.so

    there is no asset/render.so

    Ignore above cudart dlerror if you do not have a GPU set up on your machine. Traceback (most recent call last): File "demo_video.py", line 53, in main(args) File "demo_video.py", line 18, in main color = service.TrianglesMeshRender("asset/render.so", File "/home/xuhao/workspace/Dense-Head-Pose-Estimation-main/service/CtypesMeshRender.py", line 17, in init self._clibs = ctypes.CDLL(clibs)

    opened by hitxuhao 1
  • asset/render.so: cannot open shared object file: No such file or director

    asset/render.so: cannot open shared object file: No such file or director

    It worked well with: $ python3 demo_video.py -m pose -f ./re.mp4 $python3 demo_video.py -m sparse -f ./re.mp4 $ python3 demo_video.py -m dense -f ./re.mp4

    but with mesh not as you see here:

    $python3 demo_video.py -m mesh -f ./re.mp4 2022-12-05 17:03:03.713207: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1 Traceback (most recent call last): File "demo_video.py", line 55, in main(args) File "demo_video.py", line 21, in main color = service.TrianglesMeshRender("asset/render.so", "asset/triangles.npy") File "/home/redhwan/2/HPE/tensorflow/Dense-Head-Pose-Estimation-main/service/CtypesMeshRender.py", line 17, in init self._clibs = ctypes.CDLL(clibs) File "/usr/lib/python3.8/ctypes/init.py", line 373, in init self._handle = _dlopen(self._name, mode) OSError: asset/render.so: cannot open shared object file: No such file or directory

    opened by Algabri 0
  • Memory usage

    Memory usage

    hi @1996scarlet

    Is it expected that the model uses around 18GB of video RAM? Or does it maximises the RAM available to help with speed? (I got a 24GB ram GPU, and usage goes up to 22.2GB while inferencing) Thanks for this amazing MIT license repo btw.

    opened by Tetsujinfr 0
Releases(v1.0)
  • v1.0(Feb 1, 2021)

    Stable Head Pose Estimation and Landmark Regression via 3D Dense Face Reconstruction

    • CPU real-time face deteciton, alignment, and reconstruction pipeline.
    • Lightweight render library, 5x faster (3ms vs 15ms) than the Sim3DR tools.
    • Camera matrix and dense/sparse landmarks prediction via a single network.
    • Generate facial parameters for robust head pose and expression estimation.
    Source code(tar.gz)
    Source code(zip)
Owner
Remilia Scarlet
Remilia Scarlet
Skipgram Negative Sampling in PyTorch

PyTorch SGNS Word2Vec's SkipGramNegativeSampling in Python. Yet another but quite general negative sampling loss implemented in PyTorch. It can be use

Jamie J. Seol 287 Dec 14, 2022
Neurolab is a simple and powerful Neural Network Library for Python

Neurolab Neurolab is a simple and powerful Neural Network Library for Python. Contains based neural networks, train algorithms and flexible framework

152 Dec 06, 2022
Codes for ACL-IJCNLP 2021 Paper "Zero-shot Fact Verification by Claim Generation"

Zero-shot-Fact-Verification-by-Claim-Generation This repository contains code and models for the paper: Zero-shot Fact Verification by Claim Generatio

Liangming Pan 47 Jan 01, 2023
An open-source online reverse dictionary.

An open-source online reverse dictionary.

THUNLP 6.3k Jan 09, 2023
The official repository for paper ''Domain Generalization for Vision-based Driving Trajectory Generation'' submitted to ICRA 2022

DG-TrajGen The official repository for paper ''Domain Generalization for Vision-based Driving Trajectory Generation'' submitted to ICRA 2022. Our Meth

Wang 25 Sep 26, 2022
DockStream: A Docking Wrapper to Enhance De Novo Molecular Design

DockStream Description DockStream is a docking wrapper providing access to a collection of ligand embedders and docking backends. Docking execution an

AstraZeneca - Molecular AI 72 Jan 02, 2023
Regularizing Generative Adversarial Networks under Limited Data (CVPR 2021)

Regularizing Generative Adversarial Networks under Limited Data [Project Page][Paper] Implementation for our GAN regularization method. The proposed r

Google 148 Nov 18, 2022
code for Grapadora research paper experimentation

Road feature embedding selection method Code for research paper experimentation Abstract Traffic forecasting models rely on data that needs to be sens

Eric López Manibardo 0 May 26, 2022
Prevent `CUDA error: out of memory` in just 1 line of code.

🐨 Koila Koila solves CUDA error: out of memory error painlessly. Fix it with just one line of code, and forget it. 🚀 Features 🙅 Prevents CUDA error

RenChu Wang 1.7k Jan 02, 2023
Pipeline code for Sequential-GAM(Genome Architecture Mapping).

Sequential-GAM Pipeline code for Sequential-GAM(Genome Architecture Mapping). mapping whole_preprocess.sh include the whole processing of mapping. usa

3 Nov 03, 2022
Real-Time and Accurate Full-Body Multi-Person Pose Estimation&Tracking System

News! Aug 2020: v0.4.0 version of AlphaPose is released! Stronger tracking! Include whole body(face,hand,foot) keypoints! Colab now available. Dec 201

Machine Vision and Intelligence Group @ SJTU 6.7k Dec 28, 2022
Lecture materials for Cornell CS5785 Applied Machine Learning (Fall 2021)

Applied Machine Learning (Cornell CS5785, Fall 2021) This repo contains executable course notes and slides for the Applied ML course at Cornell and Co

Volodymyr Kuleshov 103 Dec 31, 2022
AirPose: Multi-View Fusion Network for Aerial 3D Human Pose and Shape Estimation

AirPose AirPose: Multi-View Fusion Network for Aerial 3D Human Pose and Shape Estimation Check the teaser video This repository contains the code of A

Robot Perception Group 41 Dec 05, 2022
Rule Based Classification Project

Kural Tabanlı Sınıflandırma ile Potansiyel Müşteri Getirisi Hesaplama İş Problemi: Bir oyun şirketi müşterilerinin bazı özelliklerini kullanaraknseviy

Şafak 1 Jan 12, 2022
GAN encoders in PyTorch that could match PGGAN, StyleGAN v1/v2, and BigGAN. Code also integrates the implementation of these GANs.

MTV-TSA: Adaptable GAN Encoders for Image Reconstruction via Multi-type Latent Vectors with Two-scale Attentions. This is the official code release fo

owl 37 Dec 24, 2022
Studying Python release adoptions by looking at PyPI downloads

Analysis of version adoptions on PyPI We get PyPI download statistics via Google's BigQuery using the pypinfo tool. Usage First you need to get an acc

Julien Palard 9 Nov 04, 2022
[CVPRW 2021] Code for Region-Adaptive Deformable Network for Image Quality Assessment

RADN [CVPRW 2021] Code for Region-Adaptive Deformable Network for Image Quality Assessment [Paper on arXiv] Overview Update [2021/5/7] add codes for W

IIGROUP 53 Dec 28, 2022
Single-Stage 6D Object Pose Estimation, CVPR 2020

Overview This repository contains the code for the paper Single-Stage 6D Object Pose Estimation. Yinlin Hu, Pascal Fua, Wei Wang and Mathieu Salzmann.

CVLAB @ EPFL 89 Dec 26, 2022
Learn other languages ​​using artificial intelligence with python.

The main idea of ​​the project is to facilitate the learning of other languages. We created a simple AI that will interact with you. Just ask questions that if she knows, she will answer.

Pedro Rodrigues 2 Jun 07, 2022
[ICLR 2021] Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments.

[ICLR 2021] RAPID: A Simple Approach for Exploration in Reinforcement Learning This is the Tensorflow implementation of ICLR 2021 paper Rank the Episo

Daochen Zha 48 Nov 21, 2022