Adversarial Framework for (non-) Parametric Image Stylisation Mosaics

Overview

Fully Adversarial Mosaics (FAMOS)

Pytorch implementation of the paper "Copy the Old or Paint Anew? An Adversarial Framework for (non-) Parametric Image Stylization" available at http://arxiv.org/abs/1811.09236.

This code allows to generate image stylisation using an adversarial approach combining parametric and non-parametric elements. Tested to work on Ubuntu 16.04, Pytorch 0.4, Python 3.6. Nvidia GPU p100. It is recommended to have a GPU with 12, 16GB, or more of VRAM.

Parameters

Our method has many possible settings. You can specify them with command-line parameters. The options parser that defines these parameters is in the config.py file and the options are parsed there. You are free to explore them and discover the functionality of FAMOS, which can cover a very broad range of image stylization settings.

There are 5 groups of parameter types:

  • data path and loading parameters
  • neural network parameters
  • regularization and loss criteria weighting parameters
  • optimization parameters
  • parameters of the stochastic noise -- see PSGAN

Update Febr. 2019: video frame-by-frame rendering supported

mosaicGAN.py can now render a whole folder of test images with the trained model. Example videos: lion video with Münich and Berlin

Just specify

python mosaicGAN.py --texturePath=samples/milano/ --contentPath=myFolder/ --testImage=myFolder/ 

with your myFolder and all images from that folder will be rendered by the generator of the GAN. Best to use the same test folder as content folder for training. To use in a video editing pipeline, save all video frames as images with a tool like AVIDEMUX, train FAMOS and save rendered frames, assemble again as video. Note: this my take some time to render thousands of images, you can edit in the code VIDEO_SAVE_FREQ to render the test image folder less frequently.

Update Jan. 2019: new functionality for texture synthesis

Due to interest in a new Pytorch implementation of our last paper "Texture Synthesis with Spatial Generative Adversarial Networks" (PSGAN) we added a script reimplementing it in the current repository. It shares many components with the texture mosaic stylization approach. A difference: PSGAN has no content image and loss, the generator is conditioned only on noise. Example call for texture synthesis:

python PSGAN.py --texturePath=samples/milano/ --ngf=120 --zLoc=50 --ndf=120 --nDep=5 --nDepD=5 --batchSize=16

In general, texture synthesis is much faster than the other methods in this repository, so feel free to add more channels and increase th batchsize. For more details and inspiration how to play with texture synthesis see our old repository with Lasagne code for PSGAN.

Usage: parametric convolutional adversarial mosaic

We provide scripts that have a main loop in which we (i) train an adversarial stylization model and (ii) save images (inference mode). If you need it, you can easily modify the code to save a trained model and load it later to do inference on many other images, see comments at the end of mosaicGAN.py.

In the simplest case, let us start an adversarial mosaic using convolutional networks. All you need is to specify the texture and content folders:

python mosaicGAN.py --texturePath=samples/milano/ --contentPath=samples/archimboldo/

This repository includes sample style files (4 satellite views of Milano, from Google Maps) and a portrait of Archimboldo (from the Google Art Project). Our GAN method will start running and training, occasionally saving results in "results/milano/archimboldo/" and printing the loss values to the terminal. Note that we use the first image found in contentPath as the default full-size output image stylization from FAMOS. You can also specify another image file name testImage to do out-of-sample stylization (inference).

This version uses DCGAN by default, which works nicely for the convolutional GAN we have here. Add the parameter LS for a least squares loss, which also works nicely. Interestingly, WGAN-GP is poorer for our model, which we did not investigate in detail.

If you want to tune the optimisation and model, you can adjust the layers and channels of the Generator and Discriminator, and also choose imageSize and batchSize. All this will effect the speed and performance of the model. You can also tweak the correspondance map cLoss and the content loss weighting fContent

python mosaicGAN.py --texturePath=samples/milano/ --contentPath=samples/archimboldo/ --imageSize=192 --batchSize=8 --ngf=80 --ndf=80  --nDepD=5  --nDep=4 --cLoss=101 --fContent=.6

Other interesting options are skipConnections and Ubottleneck. By disabling the skip connections of the Unet and defining a smaller bottleneck we can reduce the effect of the content image and emphasize more the texture style of the output.

Usage: the full FAMOS approach with parametric and non-parametric aspects

Our method has the property of being able to copy pixels from template images together with the convolutional generation of the previous section.

python mosaicFAMOS.py  --texturePath=samples/milano/ --contentPath=samples/archimboldo/ --N=80 --mirror=True --dIter=2 --WGAN=True

Here we specify N=80 memory templates to copy from. In addition, we use mirror augmentation to get nice kaleidoscope-like effects in the template (and texture distribution). We use the WGAN GAN criterium, which works better for the combined parametric/non-parametric case (experimenting with the usage of DCGAN and WGAN depending on the architecture is advised). We set to use dIter=2 D steps for each G step.

The code also supports a slightly more complicated implementation than the one described in the paper. By setting multiScale=True a mixed template of images I_M on multiple levels of the Unet is used. In addition, by setting nBlocks=2 we will add residual layers to the decoder of the Unet, for a model with even higher capacity. Finally, you can also set refine=True and add a second Unet to refine the results of the first one. Of course, all these additional layers come at a computational cost -- selecting the layer depth, channel width, and the use of all these additional modules is a matter of trade-off and experimenting.

python mosaicFAMOS.py  --texturePath=samples/milano/ --contentPath=samples/archimboldo/ --N=80 --mirror=True --multiScale=True --nBlocks=1 --dIter=2 --WGAN=True

The method will save mosaics occasionally, and optionally you can specify a testImage (size smaller than the initial content image) to check out-of-sample performance. You can check the patches image saved regularly how the patch based training proceeds. The files has a column per batch-instance, and 6 rows showing the quantities from the paper:

  • I_C content patch
  • I_M mixed template patch on highest scale
  • I_G parametric generation component
  • I blended patch
  • \alpha blending mask
  • A mixing matrix

License

Please make sure to cite/acknowledge our paper, if you use any of the contained code in your own projects or publication.

The MIT License (MIT)

Copyright © 2018 Zalando SE, https://tech.zalando.com

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Owner
Zalando Research
Repositories of the research branch of Zalando SE
Zalando Research
A collection of random and hastily hacked together scripts for investigating EU-DCC

A collection of random and hastily hacked together scripts for investigating EU-DCC

Ryan Barrett 8 Mar 01, 2022
Relative Uncertainty Learning for Facial Expression Recognition

Relative Uncertainty Learning for Facial Expression Recognition The official implementation of the following paper at NeurIPS2021: Title: Relative Unc

35 Dec 28, 2022
K-Means Clustering and Hierarchical Clustering Unsupervised Learning Solution in Python3.

Unsupervised Learning - K-Means Clustering and Hierarchical Clustering - The Heritage Foundation's Economic Freedom Index Analysis 2019 - By David Sal

David Salako 1 Jan 12, 2022
Official Code for "Non-deep Networks"

Non-deep Networks arXiv:2110.07641 Ankit Goyal, Alexey Bochkovskiy, Jia Deng, Vladlen Koltun Overview: Depth is the hallmark of DNNs. But more depth m

Ankit Goyal 567 Dec 12, 2022
External Attention Network

Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks paper : https://arxiv.org/abs/2105.02358 Jittor code will come soon

MenghaoGuo 357 Dec 11, 2022
Unsupervised Feature Ranking via Attribute Networks.

FRANe Unsupervised Feature Ranking via Attribute Networks (FRANe) converts a dataset into a network (graph) with nodes that correspond to the features

7 Sep 29, 2022
PyTorch implementation of SIFT descriptor

This is an differentiable pytorch implementation of SIFT patch descriptor. It is very slow for describing one patch, but quite fast for batch. It can

Dmytro Mishkin 150 Dec 24, 2022
Algo-burn - Script to configure an Algorand address as a "burn" address for one or more ASA tokens

Algorand Burn Address This is a simple script to illustrate how a "burn address"

GSD 5 May 10, 2022
A simple rest api serving a deep learning model that classifies human gender based on their faces. (vgg16 transfare learning)

this is a simple rest api serving a deep learning model that classifies human gender based on their faces. (vgg16 transfare learning)

crispengari 5 Dec 09, 2021
Code for "LASR: Learning Articulated Shape Reconstruction from a Monocular Video". CVPR 2021.

LASR Installation Build with conda conda env create -f lasr.yml conda activate lasr # install softras cd third_party/softras; python setup.py install;

Google 157 Dec 26, 2022
Fashion Recommender System With Python

Fashion-Recommender-System Thr growing e-commerce industry presents us with a la

Omkar Gawade 2 Feb 02, 2022
Official code for 'Pixel-wise Energy-biased Abstention Learning for Anomaly Segmentationon Complex Urban Driving Scenes'

PEBAL This repo contains the Pytorch implementation of our paper: Pixel-wise Energy-biased Abstention Learning for Anomaly Segmentationon Complex Urba

Yu Tian 115 Dec 29, 2022
TorchOk - The toolkit for fast Deep Learning experiments in Computer Vision

TorchOk - The toolkit for fast Deep Learning experiments in Computer Vision

52 Dec 23, 2022
Statsmodels: statistical modeling and econometrics in Python

About statsmodels statsmodels is a Python package that provides a complement to scipy for statistical computations including descriptive statistics an

statsmodels 8.1k Jan 02, 2023
Code for the paper "Curriculum Dropout", ICCV 2017

Curriculum Dropout Dropout is a very effective way of regularizing neural networks. Stochastically "dropping out" units with a certain probability dis

Pietro Morerio 21 Jan 02, 2022
PlenOctree Extraction algorithm

PlenOctrees_NeRF-SH This is an implementation of the Paper PlenOctrees for Real-time Rendering of Neural Radiance Fields. Not only the code provides t

49 Nov 05, 2022
RMTD: Robust Moving Target Defence Against False Data Injection Attacks in Power Grids

RMTD: Robust Moving Target Defence Against False Data Injection Attacks in Power Grids Real-time detection performance. This repo contains the code an

0 Nov 10, 2021
Video Matting via Consistency-Regularized Graph Neural Networks

Video Matting via Consistency-Regularized Graph Neural Networks Project Page | Real Data | Paper Installation Our code has been tested on Python 3.7,

41 Dec 26, 2022
Exploration & Research into cross-domain MEV. Initial focus on ETH/POLYGON.

xMEV, an apt exploration This is a small exploration on the xMEV opportunities between Polygon and Ethereum. It's a data analysis exercise on a few pa

odyslam.eth 7 Oct 18, 2022
PyTorch implementation of HDN(Homography Decomposition Networks) for planar object tracking

Homography Decomposition Networks for Planar Object Tracking This project is the offical PyTorch implementation of HDN(Homography Decomposition Networ

CaptainHook 48 Dec 15, 2022