PIXIE: Collaborative Regression of Expressive Bodies

Related tags

Deep LearningPIXIE
Overview

PIXIE: Collaborative Regression of Expressive Bodies

[Project Page]

This is the official Pytorch implementation of PIXIE.

PIXIE reconstructs an expressive body with detailed face shape and hand articulation from a single image. PIXIE does this by regressing the body, face and hands directly from image pixels using a neural network that includes a novel moderator, which attends to add weights information about the different body parts. Unlike prior work, PIXIE estimates bodies with a gender-appropriate shape but does so in a gender neutral shape space to accommodate non-binary shapes. Please refer to the Paper for more details.

The main features of PIXIE are:

  • Expressive body estimation: Given a single image, PIXIE reconstructs the 3D body shape and pose, hand articulation and facial expression as SMPL-X parameters
  • Facial details: PIXIE extracts detailed face shape, including wrinkles, using DECA
  • Facial texture: PIXIE also returns a estimate of the albedo of the subject
  • Animation: The estimated body can be re-posed and animated
  • Robust: Tested on full-body images in unconstrained conditions. The moderation strategy prevents unnatural poses. Overall, our method is robust to: various poses, illumination conditions and occlusions
  • Accurate: state-of-the-art expressive body reconstruction
  • Fast: this is a direct regression method (pixels in, SMPL-X out)

Getting started

Please follow the installation instructions to install all necessary packages and download the data.

Demo

Expressive 3D body reconstruction

python demos/demo_fit_body.py --saveObj True 

This return the estimated 3D body geometry with texture, in the form of an obj file, and render it from multiple viewpoints. If you set the optional --deca_path argument then the result will also contain facial details from DECA, provided that the face moderator is confident enough. Please run python demos/demo_fit_body.py --help for a more detailed description of the various available options.

input body image, estimated 3D body, with facial details, with texture, different views

3D face reconstruction

python demos/demo_fit_face.py --saveObj True --showBody True

Note that, given only a face image, our method still regresses the full SMPL-X parameters, producing a body mesh (as shown in the rightmost image). Futher, note how different face shapes produce different body shapes. The face tells us a lot about the body.

input face image, estimated face, with facial details, with texture, whole body in T-pose

3D hand reconstruction

python demos/demo_fit_hand.py --saveObj True

We do not provide support for hand detection, please make sure that to pass hand-only images and flip horizontally all left hands.

input hand image, estimated hand, with texture(fixed texture).

Animation

python demos/demo_animate_body.py 

Bodies estimated by PIXIE are easily animated. For example, we can estimate the body from one image and animate with the poses regressed from a different image sequence.

The visualization contains the input image, the predicted expressive 3D body, the animation result, the reference video and its corresponding reconstruction. For the latter, the color of the hands and head represents the confidence of the corresponding moderators. A lighter color means that PIXIE trusts more the information of the body image rather than the parts, which can happen when a person is facing away from the camera for example.

Notes

You can find more details on our method, as well as a discussion of the limitations of PIXIE here.

Citation

If you find our work useful to your research, please consider citing:

@inproceedings{PIXIE:2021,
      title={Collaborative Regression of Expressive Bodies using Moderation}, 
      author={Yao Feng and Vasileios Choutas and Timo Bolkart and Dimitrios Tzionas and Michael J. Black},
      booktitle={International Conference on 3D Vision (3DV)},
      year={2021}
}

License

This code and model are available for non-commercial scientific research purposes as defined in the LICENSE file. By downloading and using the code and model you agree to the terms in the LICENSE.

Acknowledgments

For functions or scripts that are based on external sources, we acknowledge the origin individually in each file.
Here are some great resources we benefit from:

We would also like to thank the authors of other public body regression methods, which allow us to easily perform quantitative and qualitative comparisons:
HMR, SPIN, frankmocap

Last but not least, we thank Victoria Fernández Abrevaya, Yinghao Huang and Radek Danecek for their helpful comments and proof reading, and Yuliang Xiu for his help in capturing demo sequences. This research was partially supported by the Max Planck ETH Center for Learning Systems. Some of the images used in the qualitative examples come from pexels.com.

Contact

For questions, please contact [email protected].
For commercial licensing (and all related questions for business applications), please contact [email protected].

Owner
Yao Feng
Yao Feng
[NeurIPS'20] Self-supervised Co-Training for Video Representation Learning. Tengda Han, Weidi Xie, Andrew Zisserman.

CoCLR: Self-supervised Co-Training for Video Representation Learning This repository contains the implementation of: InfoNCE (MoCo on videos) UberNCE

Tengda Han 271 Jan 02, 2023
Basit bir burç modülü.

Bu modulu burclar hakkinda gundelik bir sekilde bilgi alin diye yaptim ve sizler icin kullanima sunuyorum. Modulun kullanimi asiri basit: Ornek Kullan

Special 17 Jun 08, 2022
Hub is a dataset format with a simple API for creating, storing, and collaborating on AI datasets of any size.

Hub is a dataset format with a simple API for creating, storing, and collaborating on AI datasets of any size. The hub data layout enables rapid transformations and streaming of data while training m

Activeloop 5.1k Jan 08, 2023
The code for "Deep Level Set for Box-supervised Instance Segmentation in Aerial Images".

Deep Levelset for Box-supervised Instance Segmentation in Aerial Images Wentong Li, Yijie Chen, Wenyu Liu, Jianke Zhu* This code is based on MMdetecti

sunshine.lwt 112 Jan 05, 2023
Visual Adversarial Imitation Learning using Variational Models (VMAIL)

Visual Adversarial Imitation Learning using Variational Models (VMAIL) This is the official implementation of the NeurIPS 2021 paper. Project website

14 Nov 18, 2022
基于tensorflow 2.x的图片识别工具集

Classification.tf2 基于tensorflow 2.x的图片识别工具集 功能 粗粒度场景图片分类 细粒度场景图片分类 其他场景图片分类 模型部署 tensorflow serving本地推理和docker部署 tensorRT onnx ... 数据集 https://hyper.a

Wei Qi 1 Nov 03, 2021
PyTorch implementation of Graph Convolutional Networks in Feature Space for Image Deblurring and Super-resolution, IJCNN 2021.

GCResNet PyTorch implementation of Graph Convolutional Networks in Feature Space for Image Deblurring and Super-resolution, IJCNN 2021. The code will

11 May 19, 2022
DrQ-v2: Improved Data-Augmented Reinforcement Learning

DrQ-v2: Improved Data-Augmented RL Agent Method DrQ-v2 is a model-free off-policy algorithm for image-based continuous control. DrQ-v2 builds on DrQ,

Facebook Research 234 Jan 01, 2023
Implementation of PersonaGPT Dialog Model

PersonaGPT An open-domain conversational agent with many personalities PersonaGPT is an open-domain conversational agent cpable of decoding personaliz

ILLIDAN Lab 42 Jan 01, 2023
PyTorch code of my WACV 2022 paper Improving Model Generalization by Agreement of Learned Representations from Data Augmentation

Improving Model Generalization by Agreement of Learned Representations from Data Augmentation (WACV 2022) Paper ArXiv Why it matters? When data augmen

Rowel Atienza 5 Mar 04, 2022
CCNet: Criss-Cross Attention for Semantic Segmentation (TPAMI 2020 & ICCV 2019).

CCNet: Criss-Cross Attention for Semantic Segmentation Paper Links: Our most recent TPAMI version with improvements and extensions (Earlier ICCV versi

Zilong Huang 1.3k Dec 27, 2022
A Python script that creates subtitles of a given length from text paragraphs that can be easily imported into any Video Editing software such as FinalCut Pro for further adjustments.

Text to Subtitles - Python This python file creates subtitles of a given length from text paragraphs that can be easily imported into any Video Editin

Dmytro North 9 Dec 24, 2022
Implementation of Diverse Semantic Image Synthesis via Probability Distribution Modeling

Diverse Semantic Image Synthesis via Probability Distribution Modeling (CVPR 2021) Paper Zhentao Tan, Menglei Chai, Dongdong Chen, Jing Liao, Qi Chu,

tzt 45 Nov 17, 2022
Configure SRX interfaces with Scrapli

Configure SRX interfaces with Scrapli Overview This example will show how to configure interfaces on Juniper's SRX firewalls. In addition to the Pytho

Calvin Remsburg 1 Jan 07, 2022
Official implementation of GraphMask as presented in our paper Interpreting Graph Neural Networks for NLP With Differentiable Edge Masking.

GraphMask This repository contains an implementation of GraphMask, the interpretability technique for graph neural networks presented in our ICLR 2021

Michael Schlichtkrull 29 Sep 02, 2022
A tool to prepare websites grabbed with wget for local viewing.

makelocal A tool to prepare websites grabbed with wget for local viewing. exapmples After fetching xkcd.com with: wget -r -no-remove-listing -r -N --p

5 Apr 23, 2022
Code for technical report "An Improved Baseline for Sentence-level Relation Extraction".

RE_improved_baseline Code for technical report "An Improved Baseline for Sentence-level Relation Extraction". Requirements torch = 1.8.1 transformers

Wenxuan Zhou 74 Nov 29, 2022
CrossNorm and SelfNorm for Generalization under Distribution Shifts (ICCV 2021)

CrossNorm (CN) and SelfNorm (SN) (Accepted at ICCV 2021) This is the official PyTorch implementation of our CNSN paper, in which we propose CrossNorm

100 Dec 28, 2022
scikit-learn inspired API for CRFsuite

sklearn-crfsuite sklearn-crfsuite is a thin CRFsuite (python-crfsuite) wrapper which provides interface simlar to scikit-learn. sklearn_crfsuite.CRF i

417 Dec 20, 2022
Official implementation of TMANet.

Temporal Memory Attention for Video Semantic Segmentation, arxiv Introduction We propose a Temporal Memory Attention Network (TMANet) to adaptively in

wanghao 94 Dec 02, 2022