an implementation of 3D Ken Burns Effect from a Single Image using PyTorch

Overview

3d-ken-burns

This is a reference implementation of 3D Ken Burns Effect from a Single Image [1] using PyTorch. Given a single input image, it animates this still image with a virtual camera scan and zoom subject to motion parallax. Should you be making use of our work, please cite our paper [1].

Paper

setup

Several functions are implemented in CUDA using CuPy, which is why CuPy is a required dependency. It can be installed using pip install cupy or alternatively using one of the provided binary packages as outlined in the CuPy repository. Please also make sure to have the CUDA_HOME environment variable configured.

In order to generate the video results, please also make sure to have pip install moviepy installed.

usage

To run it on an image and generate the 3D Ken Burns effect fully automatically, use the following command.

python autozoom.py --in ./images/doublestrike.jpg --out ./autozoom.mp4

To start the interface that allows you to manually adjust the camera path, use the following command. You can then navigate to http://localhost:8080/ and load an image using the button on the bottom right corner. Please be patient when loading an image and saving the result, there is a bit of background processing going on.

python interface.py

To run the depth estimation to obtain the raw depth estimate, use the following command. Please note that this script does not perform the depth adjustment, see #22 for information on how to add it.

python depthestim.py --in ./images/doublestrike.jpg --out ./depthestim.npy

To benchmark the depth estimation, run python benchmark-ibims.py or python benchmark-nyu.py. You can use it to easily verify that the provided implementation runs as expected.

colab

If you do not have a suitable environment to run this projects then you could give Colab a try. It allows you to run the project in the cloud, free of charge. There are several people who provide Colab notebooks that should get you started. A few that I am aware of include one from Arnaldo Gabriel, one from Vlad Alex, and one from Ahmed Harmouche.

dataset

This dataset is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Public License (CC BY-NC-SA 4.0) and may only be used for non-commercial purposes. Please see the LICENSE file for more information.

scene mode color depth normal
asdf flying 3.7 GB 1.0 GB 2.9 GB
asdf walking 3.6 GB 0.9 GB 2.7 GB
blank flying 3.2 GB 1.0 GB 2.8 GB
blank walking 3.0 GB 0.9 GB 2.7 GB
chill flying 5.4 GB 1.1 GB 10.8 GB
chill walking 5.2 GB 1.0 GB 10.5 GB
city flying 0.8 GB 0.2 GB 0.9 GB
city walking 0.7 GB 0.2 GB 0.8 GB
environment flying 1.9 GB 0.5 GB 3.5 GB
environment walking 1.8 GB 0.5 GB 3.3 GB
fort flying 5.0 GB 1.1 GB 9.2 GB
fort walking 4.9 GB 1.1 GB 9.3 GB
grass flying 1.1 GB 0.2 GB 1.9 GB
grass walking 1.1 GB 0.2 GB 1.6 GB
ice flying 1.2 GB 0.2 GB 2.1 GB
ice walking 1.2 GB 0.2 GB 2.0 GB
knights flying 0.8 GB 0.2 GB 1.0 GB
knights walking 0.8 GB 0.2 GB 0.9 GB
outpost flying 4.8 GB 1.1 GB 7.9 GB
outpost walking 4.6 GB 1.0 GB 7.4 GB
pirates flying 0.8 GB 0.2 GB 0.8 GB
pirates walking 0.7 GB 0.2 GB 0.8 GB
shooter flying 0.9 GB 0.2 GB 1.1 GB
shooter walking 0.9 GB 0.2 GB 1.0 GB
shops flying 0.2 GB 0.1 GB 0.2 GB
shops walking 0.2 GB 0.1 GB 0.2 GB
slums flying 0.5 GB 0.1 GB 0.8 GB
slums walking 0.5 GB 0.1 GB 0.7 GB
subway flying 0.5 GB 0.1 GB 0.9 GB
subway walking 0.5 GB 0.1 GB 0.9 GB
temple flying 1.7 GB 0.4 GB 3.1 GB
temple walking 1.7 GB 0.3 GB 2.8 GB
titan flying 6.2 GB 1.1 GB 11.5 GB
titan walking 6.0 GB 1.1 GB 11.3 GB
town flying 1.7 GB 0.3 GB 3.0 GB
town walking 1.8 GB 0.3 GB 3.0 GB
underland flying 5.4 GB 1.2 GB 12.1 GB
underland walking 5.1 GB 1.2 GB 11.4 GB
victorian flying 0.5 GB 0.1 GB 0.8 GB
victorian walking 0.4 GB 0.1 GB 0.7 GB
village flying 1.6 GB 0.3 GB 2.8 GB
village walking 1.6 GB 0.3 GB 2.7 GB
warehouse flying 0.9 GB 0.2 GB 1.5 GB
warehouse walking 0.8 GB 0.2 GB 1.4 GB
western flying 0.8 GB 0.2 GB 0.9 GB
western walking 0.7 GB 0.2 GB 0.8 GB

Please note that this is an updated version of the dataset that we have used in our paper. So while it has fewer scenes in total, each sample capture now has a varying focal length which should help with generalizability. Furthermore, some examples are either over- or under-exposed and it would be a good idea to remove these outliers. Please see #37, #39, and #40 for supplementary discussions.

video

Video

license

This is a project by Adobe Research. It is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Public License (CC BY-NC-SA 4.0) and may only be used for non-commercial purposes. Please see the LICENSE file for more information.

references

[1]  @article{Niklaus_TOG_2019,
         author = {Simon Niklaus and Long Mai and Jimei Yang and Feng Liu},
         title = {3D Ken Burns Effect from a Single Image},
         journal = {ACM Transactions on Graphics},
         volume = {38},
         number = {6},
         pages = {184:1--184:15},
         year = {2019}
     }

acknowledgment

The video above uses materials under a Creative Common license or with the owner's permission, as detailed at the end.

Owner
Simon Niklaus
Research Scientist at Adobe
Simon Niklaus
Exploiting Robust Unsupervised Video Person Re-identification

Exploiting Robust Unsupervised Video Person Re-identification Implementation of the proposed uPMnet. For the preprint, please refer to [Arxiv]. Gettin

1 Apr 09, 2022
Deep Residual Learning for Image Recognition

Deep Residual Learning for Image Recognition This is a Torch implementation of "Deep Residual Learning for Image Recognition",Kaiming He, Xiangyu Zhan

Kimmy 561 Dec 01, 2022
Network Compression via Central Filter

Network Compression via Central Filter Environments The code has been tested in the following environments: Python 3.8 PyTorch 1.8.1 cuda 10.2 torchsu

2 May 12, 2022
Simple PyTorch implementations of Badnets on MNIST and CIFAR10.

Simple PyTorch implementations of Badnets on MNIST and CIFAR10.

Vera 75 Dec 13, 2022
Codebase for Attentive Neural Hawkes Process (A-NHP) and Attentive Neural Datalog Through Time (A-NDTT)

Introduction Codebase for the paper Transformer Embeddings of Irregularly Spaced Events and Their Participants. This codebase contains two packages: a

Alan Yang 28 Dec 12, 2022
NanoDet-Plus⚡Super fast and lightweight anchor-free object detection model. đŸ”„Only 980 KB(int8) / 1.8MB (fp16) and run 97FPS on cellphoneđŸ”„

NanoDet-Plus⚡Super fast and lightweight anchor-free object detection model. đŸ”„Only 980 KB(int8) / 1.8MB (fp16) and run 97FPS on cellphoneđŸ”„

4.8k Jan 07, 2023
PyTorch original implementation of Cross-lingual Language Model Pretraining.

XLM NEW: Added XLM-R model. PyTorch original implementation of Cross-lingual Language Model Pretraining. Includes: Monolingual language model pretrain

Facebook Research 2.7k Dec 27, 2022
Codes for “A Deeply Supervised Attention Metric-Based Network and an Open Aerial Image Dataset for Remote Sensing Change Detection”

DSAMNet The pytorch implementation for "A Deeply-supervised Attention Metric-based Network and an Open Aerial Image Dataset for Remote Sensing Change

Mengxi Liu 41 Dec 14, 2022
Unofficial pytorch implementation of the paper "Dynamic High-Pass Filtering and Multi-Spectral Attention for Image Super-Resolution"

DFSA Unofficial pytorch implementation of the ICCV 2021 paper "Dynamic High-Pass Filtering and Multi-Spectral Attention for Image Super-Resolution" (p

2 Nov 15, 2021
A Streamlit demo demonstrating the Deep Dream technique. Adapted from the TensorFlow Deep Dream tutorial.

Streamlit Demo: Deep Dream A Streamlit demo demonstrating the Deep Dream technique. Adapted from the TensorFlow Deep Dream tutorial How to run this de

Streamlit 11 Dec 12, 2022
Code related to the manuscript "Averting A Crisis In Simulation-Based Inference"

Abstract We present extensive empirical evidence showing that current Bayesian simulation-based inference algorithms are inadequate for the falsificat

Montefiore Artificial Intelligence Research 3 Nov 14, 2022
MATLAB codes of the book "Digital Image Processing Fourth Edition" converted to Python

Digital Image Processing Python MATLAB codes of the book "Digital Image Processing Fourth Edition" converted to Python TO-DO: Refactor scripts, curren

Merve Noyan 24 Oct 16, 2022
Advanced Signal Processing Notebooks and Tutorials

Advanced Digital Signal Processing Notebooks and Tutorials Prof. Dr. -Ing. Gerald Schuller Jupyter Notebooks and Videos: Renato Profeta Applied Media

Guitars.AI 115 Dec 13, 2022
Memory Efficient Attention (O(sqrt(n)) for Jax and PyTorch

Memory Efficient Attention This is unofficial implementation of Self-attention Does Not Need O(n^2) Memory for Jax and PyTorch. Implementation is almo

Amin Rezaei 126 Dec 27, 2022
Morphable Detector for Object Detection on Demand

Morphable Detector for Object Detection on Demand (ICCV 2021) PyTorch implementation of the paper Morphable Detector for Object Detection on Demand. I

9 Feb 23, 2022
FinGAT: A Financial Graph Attention Networkto Recommend Top-K Profitable Stocks

FinGAT: A Financial Graph Attention Networkto Recommend Top-K Profitable Stocks This is our implementation for the paper: FinGAT: A Financial Graph At

Yu-Che Tsai 64 Dec 13, 2022
Novel Instances Mining with Pseudo-Margin Evaluation for Few-Shot Object Detection

Novel Instances Mining with Pseudo-Margin Evaluation for Few-Shot Object Detection (NimPme) The official implementation of Novel Instances Mining with

12 Sep 08, 2022
Full-featured Decision Trees and Random Forests learner.

CID3 This is a full-featured Decision Trees and Random Forests learner. It can save trees or forests to disk for later use. It is possible to query tr

Alejandro Penate-Diaz 3 Aug 15, 2022
DvD-TD3: Diversity via Determinants for TD3 version

DvD-TD3: Diversity via Determinants for TD3 version The implementation of paper Effective Diversity in Population Based Reinforcement Learning. Instal

3 Feb 11, 2022
Global Pooling, More than Meets the Eye: Position Information is Encoded Channel-Wise in CNNs, ICCV 2021

Global Pooling, More than Meets the Eye: Position Information is Encoded Channel-Wise in CNNs, ICCV 2021 Global Pooling, More than Meets the Eye: Posi

Md Amirul Islam 32 Apr 24, 2022