Code for ICCV2021 paper SPEC: Seeing People in the Wild with an Estimated Camera

Last update: Dec 26, 2022

Overview

SPEC: Seeing People in the Wild with an Estimated Camera [ICCV 2021]

SPEC: Seeing People in the Wild with an Estimated Camera,
Muhammed Kocabas, Chun-Hao Paul Huang, Joachim Tesch, Lea Müller, Otmar Hilliges, Michael J. Black,
International Conference on Computer Vision (ICCV), 2021

Features

SPEC is a camera-aware human body pose and shape estimation method. It both predicts the camera parameters and SMPL body model for a given image. CamCalib predicts the camera parameters. SPEC uses these parameters to predict SMPL body model parameters.

This implementation:

has the demo code for SPEC and CamCalib implemented in PyTorch.
achieves SOTA results in SPEC-SYN and SPEC-MTP datasets.
shows how to perform evaluation on SPEC-SYN and SPEC-MTP datasets.

Updates

13/10/2021: Demo and evaluation code is released.

Getting Started

SPEC has been implemented and tested on Ubuntu 18.04 with python >= 3.7. If you don't have a suitable device, try running our Colab demo.

Clone the repo:

git clone https://github.com/mkocabas/SPEC.git

Install the requirements using virtualenv or conda:

# pip
source scripts/install_pip.sh

# conda
source scripts/install_conda.sh

Running the Demo

SPEC

First, you need to download the required data (i.e our trained model and SMPL model parameters). It is approximately 1GB. To do this you can just run:

source scripts/prepare_data.sh

Then, running the demo is as simple as:

python scripts/spec_demo.py \
  --image_folder data/sample_images \
  --output_folder logs/spec/sample_images

Sample demo output:

Here the green line is the horizon obtained using estimated camera parameters. On the right, the ground plane is visualized to show how accurate the global translation is.

CamCalib

If you are only interested in estimating the camera parameters of an image, run the CamCalib demo:

python scripts/camcalib_demo.py \
  --img_folder <input image folder> \
  --out_folder <output folder> \
  --show # visualize the raw network predictions

This script outputs a pickle file which contains the predicted camera parameters for each input image along with an output image which visualizes the camera parameters as a horizon line. Pickle file contains:

'vfov' : vertical field of view in radians
'f_pix': focal length in pixels
'pitch': pitch in radians
'roll' : roll in radians

Google Colab

Training

Training instructions will follow soon.

Datasets

Pano360, SPEC-MTP, and SPEC-SYN are new datasets introduced in our paper. You can download them from the Downloads section of our project page.

For Pano360 dataset, we have released the Flickr image ids which can be used to download images using FlickrAPI. We have provided a download script in this repo. Some of the images will be missing due to users deleting their photos. In this case, you can also use scrape_and_download function provided in the script to find and download more photos.

After downloading the SPEC-SYN, SPEC-MTP, Pano360, and 3DPW datasets, the data folder should look like:

data/
├── body_models
│   └── smpl
├── camcalib
│   └── checkpoints
├── dataset_extras
├── dataset_folders
│   ├── 3dpw
│   ├── pano360
│   ├── spec-mtp
│   └── spec-syn
├── sample_images
└── spec
    └── checkpoints

Evaluation

You can evaluate SPEC on SPEC-SYN, SPEC-MTP, and 3DPW datasets by running:

python scripts/spec_eval.py \
  --cfg data/spec/checkpoints/spec_config.yaml \
  --opts DATASET.VAL_DS spec-syn_spec-mtp_3dpw-test-cam

Running this script should give results reported in this table:

	W-MPJPE	PA-MPJPE	W-PVE
SPEC-MTP	124.3	71.8	147.1
SPEC-SYN	74.9	54.5	90.5
3DPW	106.7	53.3	124.7

Citation

@inproceedings{SPEC:ICCV:2021,
  title = {{SPEC}: Seeing People in the Wild with an Estimated Camera},
  author = {Kocabas, Muhammed and Huang, Chun-Hao P. and Tesch, Joachim and M\"uller, Lea and Hilliges, Otmar and Black, Michael J.},
  booktitle = {Proc. International Conference on Computer Vision (ICCV)},
  pages = {11035--11045},
  month = oct,
  year = {2021},
  doi = {},
  month_numeric = {10}
}

License

This code is available for non-commercial scientific research purposes as defined in the LICENSE file. By downloading and using this code you agree to the terms in the LICENSE. Third-party datasets and software are subject to their respective licenses.

References

We indicate if a function or script is borrowed externally inside each file. Here are some great resources we benefit:

Pano360 dataset preprocessing is borrowed from Single View Metrology in the Wild.
Most of the utility functions depends on our another paper PARE.

Consider citing these works if you use them in your project.

Contact

For questions, please contact [email protected]

For commercial licensing (and all related questions for business applications), please contact [email protected].

Comments

Translation of the camera

I'd like to know if it's possible to get a translation of the camera. I've found the code just for rotation and focal length estimation.

On this image https://github.com/mkocabas/SPEC/blob/master/docs/assets/spec_gif.gif the camera is placed in the world space somehow. How did you do this? Thanks.

opened by Dene33 4
Segmentation fault (core dumped)

Hi~ When I run the command python scripts/spec_demo.py --batch_size 1 --image_folder data/sample_images --output_folder logs/spec/sample_images, I got the error as this: Segmentation fault (core dumped) I have no idea about it, could you help me, please?

opened by JinkaiZheng 2
There is no flickr_photo_ids.npy

Hello author:

After unzipping the spec-github-data.zip, there is no data/dataset_folders/pano360/flickr_photo_ids.npy. I don't know whether I find the right place, could you help me?

Best regards.

opened by songxujay 1

video demo

It's a great job. Is the video demo not available now? `def main(args):

demo_mode = args.mode

if demo_mode == 'video':
    raise NotImplementedError
elif demo_mode == 'webcam':
    raise NotImplementedError`

opened by FatherPrime 1

Fix PARE requirement in requirement.txt

Thanks for sharing this work! Please fix the requirement in requirement.txt file of PARE. It should be (currently missing "git+" prefix): git+https://github.com/mkocabas/PARE.git

opened by Omri-CG 1
get error horizon line by camcalib_demo.py

all example_image seem get error result by camcalib_demo.py, look like this: it seems the predicted horizon line shift by model,how should I solve this problem？

opened by hhhlllyyy 0
feet fit accuracy

I've encountered frequent problems that SPEC is having to fit the model to feet correctly:

Is this the same problem as mentioned in the VIBE discussion: https://github.com/mkocabas/VIBE/issues/24#issuecomment-596714558 ? Have you tried to include the OpenPose feet keypoint predictions to the loss?

Thanks for the great work!

opened by smidm 0
Dataset npz interpretation

Dear Muhammed,

Thank you for this work and the datasets! I'm trying to interpret the data in the SPEC-SYN npz data files, but I'm not sure what each key means. Is there documentation? They are the following: ['imgname', 'center', 'scale', 'pose', 'shape', 'part', 'mmpose_keypoints', 'openpose', 'openpose_gt', 'S', 'focal_length', 'cam_rotmat', 'cam_trans', 'cam_center', 'cam_pitch', 'cam_roll', 'cam_hfov', 'cam_int', 'camcalib_pitch', 'camcalib_roll', 'camcalib_vfov', 'camcalib_f_pix']

At this point, I'd just like to plot the 24 SMPL joints on the image. Based on the array shape, I assume 'S' contains the joints. Is cam_rotmat the rotation from world space to camera space? Is the cam_trans the position of the camera or the top right part of the extrinsic matrix? I assume for this plotting exercise I can ignore everything except S, cam_rotmat, cam_trans and cam_int. Still for some reason the points end up at at very wrong places. Maybe 'S' is something else? Or am I using the camera params wrong?

Thanks! Istvan

opened by isarandi 5
Colab notebook inaccesible

Hi,I tried accesing the colab notebook but wasn't able to access it.Would you be able to kindly suggest another way to run inference as I am currently working on a windows based CPU system.

opened by sparshgarg23 0

Error during single dataset evaluation

I tried running the evaluation only for the 3DPW dataset with the following command:

python scripts/spec_eval.py --cfg data/spec/checkpoints/spec_config.yaml --opts DATASET.VAL_DS 3dpw-test-cam

But it gives the following error:

TypeError: test_step() missing 1 required positional argument: 'dataloader_nb'

Fixed it by giving a default value for dataloader_nb for the test_step function in trainer.py:

def test_step(self, batch, batch_nb, dataloader_nb=0):
    return self.validation_step(batch, batch_nb, dataloader_nb)

The error occurs because we don't append dataloader_idx to args in evaluation_loop.py:

if multiple_test_loaders or multiple_val_loaders:
    args.append(dataloader_idx)

opened by umariqb 1

Releases(v0.1)

v0.1(Oct 15, 2021)

Source code(tar.gz)
Source code(zip)
spec-github-data.zip(950.66 MB)

Owner

Muhammed Kocabas

GitHub Repository

A Tensorflow implementation of CapsNet based on Geoffrey Hinton's paper Dynamic Routing Between Capsules

CapsNet-Tensorflow A Tensorflow implementation of CapsNet based on Geoffrey Hinton's paper Dynamic Routing Between Capsules Notes: The current version

3.8k Dec 29, 2022

OcclusionFusion: realtime dynamic 3D reconstruction based on single-view RGB-D

OcclusionFusion (CVPR'2022) Project Page | Paper | Video Overview This repository contains the code for the CVPR 2022 paper OcclusionFusion, where we

193 Dec 15, 2022

(ICCV'21) Official PyTorch implementation of Relational Embedding for Few-Shot Classification

Relational Embedding for Few-Shot Classification (ICCV 2021) Dahyun Kang, Heeseung Kwon, Juhong Min, Minsu Cho [paper], [project hompage] We propose t

82 Dec 24, 2022

Code for the paper BERT might be Overkill: A Tiny but Effective Biomedical Entity Linker based on Residual Convolutional Neural Networks

Biomedical Entity Linking This repo provides the code for the paper BERT might be Overkill: A Tiny but Effective Biomedical Entity Linker based on Res

24 Oct 24, 2022

code for "Self-supervised edge features for improved Graph Neural Network training",

Self-supervised edge features for improved Graph Neural Network training Data availability: Here is a link to the raw data for the organoids dataset.

23 Dec 02, 2022

x-transformers-paddle 2.x version

x-transformers-paddle x-transformers-paddle 2.x version paddle 2.x版本 https://github.com/lucidrains/x-transformers 。 requirements paddlepaddle-gpu==2.2

7 Dec 08, 2022

Code of the paper "Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognition"

SEW (Squeezed and Efficient Wav2vec) The repo contains the code of the paper "Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speec

67 Dec 01, 2022

This is the code repository for the paper A hierarchical semantic segmentation framework for computer-vision-based bridge column damage detection

Bridge-damage-segmentation This is the code repository for the paper A hierarchical semantic segmentation framework for computer-vision-based bridge c

5 Dec 07, 2022

Temporal Segment Networks (TSN) in PyTorch

TSN-Pytorch We have released MMAction, a full-fledged action understanding toolbox based on PyTorch. It includes implementation for TSN as well as oth

1k Jan 03, 2023

Thermal Control of Laser Powder Bed Fusion using Deep Reinforcement Learning

This repository is the implementation of the paper "Thermal Control of Laser Powder Bed Fusion Using Deep Reinforcement Learning", linked here. The project makes use of the Deep Reinforcement Library

11 Dec 27, 2022

RipsNet: a general architecture for fast and robust estimation of the persistent homology of point clouds

RipsNet: a general architecture for fast and robust estimation of the persistent homology of point clouds This repository contains the code asscoiated

14 Dec 12, 2022

Code for database and frontend of webpage for Neural Fields in Visual Computing and Beyond.

Neural Fields in Visual Computing—Complementary Webpage This is based on the amazing MiniConf project from Hendrik Strobelt and Sasha Rush—thank you!

29 Nov 30, 2022

Task-related Saliency Network For Few-shot learning

Task-related Saliency Network For Few-shot learning This is an official implementation in Tensorflow of TRSN. Abstract An essential cue of human wisdo

1 Nov 18, 2021

Contrastive Learning Inverts the Data Generating Process

Official code to reproduce the results and data presented in the paper Contrastive Learning Inverts the Data Generating Process.

71 Nov 25, 2022

The official TensorFlow implementation of the paper Action Transformer: A Self-Attention Model for Short-Time Pose-Based Human Action Recognition

Action Transformer A Self-Attention Model for Short-Time Human Action Recognition This repository contains the official TensorFlow implementation of t

20 Jan 03, 2023

This repository contains the code for "Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP".

Self-Diagnosis and Self-Debiasing This repository contains the source code for Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based

62 Dec 12, 2022