This is the official repo for TransFill: Reference-guided Image Inpainting by Merging Multiple Color and Spatial Transformations at CVPR'21. According to some product reasons, we are not planning to release the training/testing codes and models. However, we will release the dataset and the scripts to prepare the dataset.

Overview

TransFill-Reference-Inpainting

This is the official repo for TransFill: Reference-guided Image Inpainting by Merging Multiple Color and Spatial Transformations (Yuqian Zhou, Connelly Barnes, Eli Shechtman, Sohrab Amirghodsi) at CVPR'21. According to some confidential reasons, we are not planning to release the training/testing codes and models. Online-demo will be public once we set up the server. However, we release the testing dataset for comparsion, and the scripts to prepare the training dataset.

[Paper] | [Project] | [Demo Video]

Introduction

Applications of TransFill: Photo Content Swap, Object Removal, Color Adjustment.

Image inpainting is the task of plausibly restoring missing pixels within a hole region that is to be removed from a target image. Most existing technologies exploit patch similarities within the image, or leverage large-scale training data to fill the hole using learned semantic and texture information. However, due to the ill-posed nature of the inpainting task, such methods struggle to complete larger holes containing complicated scenes. In this paper, we propose TransFill, a multi-homography transformed fusion method to fill the hole by referring to another source image that shares scene contents with the target image. We first align the source image to the target image by estimating multiple homographies guided by different depth levels. We then learn to adjust the color and apply a pixel-level warping to each homography-warped source image to make it more consistent with the target. Finally, a pixel-level fusion module is learned to selectively merge the different proposals. Our method achieves state-of-the-art performance on pairs of images across a variety of wide baselines and color differences, and generalizes to user-provided image pairs.

Download and Prepare RealEstate10K

We prepare the script of downloading and extracting paired frames from RealEstate10K. First, go to the RealEstate10K official website to download the .txt files. Then unzip it and put the folder into the data folder.

Run our script to download the video samples and extract paired frames with frame difference (stride) 10, 20 and 30.

python download_realestate10k.py \
--txt_dir ./data/RealEstate10K/train \
--out_dir ./RealEstate10K_frames/train \
--dataset_dir ./RealEstate10K_pair/train \
--sample_num 10

Choose the sample number to download limited number of samples (say 100 videos). You may need to install youtube-dl package or VPNs (in Mainland China) to download YouTube videos. Google also has some limitations of downloading amount, so I did not use multi-thread to increase the downloading speed on purpose. The process is fairly long, so I suggest downloading a subset of videos to extract samples first, and gradually extending it to download the whole dataset. Any other downloading issues, please inquire the original provider of RealEstate10K.

Download Testing Data

We shared the testing images in the paper, including the 'Small Set' containing 300 pairs of images from RealEstate10K, and a 'Real Set' containing 100+ challenging paired images from users. The data can be downloaded from the Google Drive.

To reproduce the results in the Table 1 of the paper, download and unzip the 'Small Set' into data folder, and run

python compute_metrics.py

The script will compare the images generated by TransFill with the ground truth images in the target folder, and return PSNR, SSIM and LPIPS score.

In the 'Real Set', ProFill and TransFill results are shared for the researchers to compare. Note that there are some failure cases within the folder, which shows the room for future works to improve TransFill.

Test on Your Own Data

We plan to set up the online demo server in the near future. But before we finish that, if you are really eager for a comparsion of the results for research purpose, feel free to send the testing data in the format of 'target', 'source', 'hole' folders to [email protected]. The resolution has better be smaller than 1K x 1K, otherwise we have to resize the image to avoid memory issues. To make fully use of the advantages of TransFill, we suggest the hole to be large enough by including more background contents of the target image.

We won't keep your data and will return the testing results to you within 2 working days.

Citation

If you think this repo and the manuscript helpful, please consider citing us.

@inproceedings{zhou2021transfill,
  title={TransFill: Reference-guided Image Inpainting by Merging Multiple Color and Spatial Transformations},
  author={Zhou, Yuqian and Barnes, Connelly and Shechtman, Eli and Amirghodsi, Sohrab},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={2266--2276},
  year={2021}
}

Acknowledgements

This project is conducted when the author interned at Adobe Photoshop and Adobe Research.

Owner
Yuqian Zhou
Ph.D of Beckman Institute, UIUC Mphil of ECE in HKUST.
Yuqian Zhou
Implementation for Simple Spectral Graph Convolution in ICLR 2021

Simple Spectral Graph Convolutional Overview This repo contains an example implementation of the Simple Spectral Graph Convolutional (S^2GC) model. Th

allenhaozhu 64 Dec 31, 2022
Expressive Power of Invariant and Equivaraint Graph Neural Networks (ICLR 2021)

Expressive Power of Invariant and Equivaraint Graph Neural Networks In this repository, we show how to use powerful GNN (2-FGNN) to solve a graph alig

Marc Lelarge 36 Dec 12, 2022
Tom-the-AI - A compound artificial intelligence software for Linux systems.

Tom the AI (version 0.82) WARNING: This software is not yet ready to use, I'm still setting up the GitHub repository. Should be ready in a few days. T

2 Apr 28, 2022
A lightweight Python-based 3D network multi-agent simulator. Uses a cell-based congestion model. Calculates risk, loudness and battery capacities of the agents. Suitable for 3D network optimization tasks.

AMAZ3DSim AMAZ3DSim is a lightweight python-based 3D network multi-agent simulator. It uses a cell-based congestion model. It calculates risk, battery

Daniel Hirsch 13 Nov 04, 2022
Implementation of Perceiver, General Perception with Iterative Attention in TensorFlow

Perceiver This Python package implements Perceiver: General Perception with Iterative Attention by Andrew Jaegle in TensorFlow. This model builds on t

Rishit Dagli 84 Oct 15, 2022
Online Multi-Granularity Distillation for GAN Compression (ICCV2021)

Online Multi-Granularity Distillation for GAN Compression (ICCV2021) This repository contains the pytorch codes and trained models described in the IC

Bytedance Inc. 299 Dec 16, 2022
Statistical and Algorithmic Investing Strategies for Everyone

Eiten - Algorithmic Investing Strategies for Everyone Eiten is an open source toolkit by Tradytics that implements various statistical and algorithmic

Tradytics 2.5k Jan 02, 2023
Reinforcement Learning Theory Book (rus)

Reinforcement Learning Theory Book (rus)

qbrick 206 Nov 27, 2022
Python implementation of a live deep learning based age/gender/expression recognizer

TUT live age estimator Python implementation of a live deep learning based age/gender/smile/celebrity twin recognizer. All components use convolutiona

Heikki Huttunen 80 Nov 21, 2022
A project to build an AI voice assistant using Python . The Voice assistant interacts with the humans to perform basic tasks.

AI_Personal_Voice_Assistant_Using_Python A project to build an AI voice assistant using Python . The Voice assistant interacts with the humans to perf

Chumui Tripura 1 Oct 30, 2021
Character Grounding and Re-Identification in Story of Videos and Text Descriptions

Character in Story Identification Network (CiSIN) This project hosts the code for our paper. Youngjae Yu, Jongseok Kim, Heeseung Yun, Jiwan Chung and

8 Dec 09, 2022
A Comprehensive Empirical Study of Vision-Language Pre-trained Model for Supervised Cross-Modal Retrieval

CLIP4CMR A Comprehensive Empirical Study of Vision-Language Pre-trained Model for Supervised Cross-Modal Retrieval The original data and pre-calculate

24 Dec 26, 2022
1st Solution For ICDAR 2021 Competition on Mathematical Formula Detection

This project releases our 1st place solution on ICDAR 2021 Competition on Mathematical Formula Detection. We implement our solution based on MMDetection, which is an open source object detection tool

yuxzho 94 Dec 25, 2022
Implementation of EMNLP 2017 Paper "Natural Language Does Not Emerge 'Naturally' in Multi-Agent Dialog" using PyTorch and ParlAI

Language Emergence in Multi Agent Dialog Code for the Paper Natural Language Does Not Emerge 'Naturally' in Multi-Agent Dialog Satwik Kottur, José M.

Karan Desai 105 Nov 25, 2022
LSTC: Boosting Atomic Action Detection with Long-Short-Term Context

LSTC: Boosting Atomic Action Detection with Long-Short-Term Context This Repository contains the code on AVA of our ACM MM 2021 paper: LSTC: Boosting

Tencent YouTu Research 9 Oct 11, 2022
PyTorch implementation for ACL 2021 paper "Maria: A Visual Experience Powered Conversational Agent".

Maria: A Visual Experience Powered Conversational Agent This repository is the Pytorch implementation of our paper "Maria: A Visual Experience Powered

Jokie 22 Dec 12, 2022
BoxInst: High-Performance Instance Segmentation with Box Annotations

Introduction This repository is the code that needs to be submitted for OpenMMLab Algorithm Ecological Challenge, the paper is BoxInst: High-Performan

88 Dec 21, 2022
Kaggle: Cell Instance Segmentation

Kaggle: Cell Instance Segmentation The goal of this challenge is to detect cells in microscope images. with simple view on how many cels have been ann

Jirka Borovec 9 Aug 12, 2022
Little Ball of Fur - A graph sampling extension library for NetworKit and NetworkX (CIKM 2020)

Little Ball of Fur is a graph sampling extension library for Python. Please look at the Documentation, relevant Paper, Promo video and External Resour

Benedek Rozemberczki 619 Dec 14, 2022
Deep Sea Treasure Environment for Multi-Objective Optimization Research

DeepSeaTreasure Environment Installation In order to get started with this environment, you can install it using the following command: python3 -m pip

imec IDLab 6 Nov 14, 2022