Official PyTorch Implementation of paper "NeLF: Neural Light-transport Field for Single Portrait View Synthesis and Relighting", EGSR 2021.

Related tags

Text Data & NLPnelf
Overview

NeLF: Neural Light-transport Field for Single Portrait View Synthesis and Relighting

Official PyTorch Implementation of paper "NeLF: Neural Light-transport Field for Single Portrait View Synthesis and Relighting", EGSR 2021.

Tiancheng Sun1*, Kai-En Lin1*, Sai Bi2, Zexiang Xu2, Ravi Ramamoorthi1

1University of California, San Diego, 2Adobe Research

*Equal contribution

Project Page | Paper | Pretrained models | Validation data | Rendering script

Requirements

Install required packages

Make sure you have up-to-date NVIDIA drivers supporting CUDA 11.1 (10.2 could work but need to change cudatoolkit package accordingly)

Run

conda env create -f environment.yml
conda activate pixelnerf

The following packages are used:

  • PyTorch (1.7 & 1.9.0 Tested)

  • OpenCV-Python

  • matplotlib

  • numpy

  • tqdm

OS system: Ubuntu 20.04

Download CelebAMask-HQ dataset link

  1. Download the dataset

  2. Remove background with the provided masks in the dataset

  3. Downsample the dataset to 512x512

  4. Store the resulting data in [path_to_data_directory]/CelebAMask

    Following this data structure

    [path_to_data_directory] --- data --- CelebAMask --- 0.jpg
                                       |              |- 1.jpg
                                       |              |- 2.jpg
                                       |              ...
                                       |- blender_both --- sub001
                                       |                |- sub002
                                       |                ...
    
    

(Optional) Download and render FaceScape dataset link

Due to FaceScape's license, we cannot release the full dataset. Instead, we will release our rendering script.

  1. Download the dataset

  2. Install Blender link

  3. Run rendering script link

Usage

Testing

  1. Download our pretrained checkpoint and testing data. Extract the content to [path_to_data_directory]. The data structure should look like this:

    [path_to_data_directory] --- data --- CelebAMask
                              |        |- blender_both
                              |        |- blender_view
                              |        ...
                              |- data_results --- nelf_ft
                              |- data_test --- validate_0
                                            |- validate_1
                                            |- validate_2
    
  2. In arg/__init__.py, setup data path by changing base_path

  3. Run python run_test.py nelf_ft [validation_data_name] [#iteration_for_the_model]

    e.g. python run_test.py nelf_ft validate_0 500000

  4. The results are stored in [path_to_data_directory]/data_test/[validation_data_name]/results

Training

Due to FaceScape's license, we are not allowed to release the full dataset. We will use validation data to run the following example.

  1. Download our validation data. Extract the content to [path_to_data_directory]. The data structure should look like this:

    [path_to_data_directory] --- data --- CelebAMask
                              |        |- blender_both
                              |        |- blender_view
                              |        ...
                              |- data_results --- nelf_ft
                              |- data_test --- validate_0
                                            |- validate_1
                                            |- validate_2
    

    (Optional) Run rendering script and render your own data.

    Remember to change line 35~42 and line 45, 46 in arg/config_nelf_ft.py accordingly.

  2. In arg/__init__.py, setup data path by changing base_path

  3. Run python run_train.py nelf_ft

  4. The intermediate results and model checkpoints are saved in [path_to_data_directory]/data_results/nelf_ft

Configs

The following config files can be found inside arg folder

Citation

@inproceedings {sun2021nelf,
    booktitle = {Eurographics Symposium on Rendering},
    title = {NeLF: Neural Light-transport Field for Portrait View Synthesis and Relighting},
    author = {Sun, Tiancheng and Lin, Kai-En and Bi, Sai and Xu, Zexiang and Ramamoorthi, Ravi},
    year = {2021},
}
Owner
Ken Lin
Ken Lin
A design of MIDI language for music generation task, specifically for Natural Language Processing (NLP) models.

MIDI Language Introduction Reference Paper: Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions: code This

Robert Bogan Kang 3 May 25, 2022
This is the source code of RPG (Reward-Randomized Policy Gradient)

RPG (Reward-Randomized Policy Gradient) Zhenggang Tang*, Chao Yu*, Boyuan Chen, Huazhe Xu, Xiaolong Wang, Fei Fang, Simon Shaolei Du, Yu Wang, Yi Wu (

40 Nov 25, 2022
Code for our paper "Transfer Learning for Sequence Generation: from Single-source to Multi-source" in ACL 2021.

TRICE: a task-agnostic transferring framework for multi-source sequence generation This is the source code of our work Transfer Learning for Sequence

THUNLP-MT 9 Jun 27, 2022
Searching keywords in PDF file folders

keyword_searching Steps to use this Python scripts: (1)Paste this script into the file folder containing the PDF files you need to search from; (2)Thi

1 Nov 08, 2021
Ukrainian TTS (text-to-speech) using Coqui TTS

title emoji colorFrom colorTo sdk app_file pinned Ukrainian TTS 🐸 green green gradio app.py false Ukrainian TTS 📢 🤖 Ukrainian TTS (text-to-speech)

Yurii Paniv 85 Dec 26, 2022
Dé op-de-vlucht Pieton vertaler. Wereldwijd gebruikt door meer dan 1.000+ succesvolle bedrijven!

Dé op-de-vlucht Pieton vertaler. Wereldwijd gebruikt door meer dan 1.000+ succesvolle bedrijven!

Lau 1 Dec 17, 2021
An Analysis Toolkit for Natural Language Generation (Translation, Captioning, Summarization, etc.)

VizSeq is a Python toolkit for visual analysis on text generation tasks like machine translation, summarization, image captioning, speech translation

Facebook Research 409 Oct 28, 2022
This is the 25 + 1 year anniversary version of the 1995 Rachford-Rice contest

Rachford-Rice Contest This is the 25 + 1 year anniversary version of the 1995 Rachford-Rice contest. Can you solve the Rachford-Rice problem for all t

13 Sep 20, 2022
Free and Open Source Machine Translation API. 100% self-hosted, offline capable and easy to setup.

LibreTranslate Try it online! | API Docs | Community Forum Free and Open Source Machine Translation API, entirely self-hosted. Unlike other APIs, it d

3.4k Dec 27, 2022
Fuzzy String Matching in Python

FuzzyWuzzy Fuzzy string matching like a boss. It uses Levenshtein Distance to calculate the differences between sequences in a simple-to-use package.

SeatGeek 8.8k Jan 01, 2023
Findings of ACL 2021

Assessing Dialogue Systems with Distribution Distances [arXiv][code] We propose to measure the performance of a dialogue system by computing the distr

Yahui Liu 16 Feb 24, 2022
customer care chatbot made with Rasa Open Source.

Customer Care Bot Customer care bot for ecomm company which can solve faq and chitchat with users, can contact directly to team. 🛠 Features Basic E-c

Dishant Gandhi 23 Oct 27, 2022
🚀 RocketQA, dense retrieval for information retrieval and question answering, including both Chinese and English state-of-the-art models.

In recent years, the dense retrievers based on pre-trained language models have achieved remarkable progress. To facilitate more developers using cutt

475 Jan 04, 2023
TunBERT is the first release of a pre-trained BERT model for the Tunisian dialect using a Tunisian Common-Crawl-based dataset.

TunBERT is the first release of a pre-trained BERT model for the Tunisian dialect using a Tunisian Common-Crawl-based dataset. TunBERT was applied to three NLP downstream tasks: Sentiment Analysis (S

InstaDeep Ltd 72 Dec 09, 2022
Convolutional 2D Knowledge Graph Embeddings resources

ConvE Convolutional 2D Knowledge Graph Embeddings resources. Paper: Convolutional 2D Knowledge Graph Embeddings Used in the paper, but do not use thes

Tim Dettmers 586 Dec 24, 2022
LSTM based Sentiment Classification using Tensorflow - Amazon Reviews Rating

LSTM based Sentiment Classification using Tensorflow - Amazon Reviews Rating (Dataset) The dataset is from Amazon Review Data (2018)

Immanuvel Prathap S 1 Jan 16, 2022
Twitter-NLP-Analysis - Twitter Natural Language Processing Analysis

Twitter-NLP-Analysis Business Problem I got last @turk_politika 3000 tweets with

Çağrı Karadeniz 7 Mar 12, 2022
Python wrapper for Stanford CoreNLP tools v3.4.1

Python interface to Stanford Core NLP tools v3.4.1 This is a Python wrapper for Stanford University's NLP group's Java-based CoreNLP tools. It can eit

Dustin Smith 610 Sep 07, 2022
pytorch implementation of Attention is all you need

A Pytorch Implementation of the Transformer: Attention Is All You Need Our implementation is largely based on Tensorflow implementation Requirements N

230 Dec 07, 2022