Bayesian dessert for Lasagne

Overview

Gelato

Coverage Status

Bayesian dessert for Lasagne

Recent results in Bayesian statistics for constructing robust neural networks have proved that it is one of the best ways to deal with uncertainty, overfitting but still having good performance. Gelato will help to use bayes for neural networks. Library heavily relies on Theano, Lasagne and PyMC3.

Installation

  • from github (assumes bleeding edge pymc3 installed)
    # pip install git+git://github.com/pymc-devs/pymc3.git
    pip install git+https://github.com/ferrine/gelato.git
  • from source
    git clone https://github.com/ferrine/gelato
    pip install -r gelato/requirements.txt
    pip install -e gelato

Usage

I use generic approach for decorating all Lasagne at once. Thus, for using Gelato you need to replace import statements for layers only. For constructing a network you need to be the in pm.Model context environment.

Warning

  • lasagne.layers.noise is not supported
  • lasagne.layers.normalization is not supported (theano problems with default updates)
  • functions from lasagne.layers are hidden in gelato as they use Lasagne classes. Some exceptions are done for lasagne.layers.helpers. I'll try to solve the problem generically in future.

Examples

For comprehensive example of using Gelato you can reference this notebook

Life Hack

Any spec class can be used standalone so feel free to use it everywhere

References

Charles Blundell et al: "Weight Uncertainty in Neural Networks" (arXiv preprint arXiv:1505.05424)

You might also like...
Bayesian optimization in PyTorch

BoTorch is a library for Bayesian Optimization built on PyTorch. BoTorch is currently in beta and under active development! Why BoTorch ? BoTorch Prov

Safe Bayesian Optimization
Safe Bayesian Optimization

SafeOpt - Safe Bayesian Optimization This code implements an adapted version of the safe, Bayesian optimization algorithm, SafeOpt [1], [2]. It also p

Bayesian Optimization using GPflow

Note: This package is for use with GPFlow 1. For Bayesian optimization using GPFlow 2 please see Trieste, a joint effort with Secondmind. GPflowOpt GP

Code for
Code for "Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations"

Infinitely Deep Bayesian Neural Networks with SDEs This library contains JAX and Pytorch implementations of neural ODEs and Bayesian layers for stocha

(under submission) Bayesian Integration of a Generative Prior for Image Restoration
(under submission) Bayesian Integration of a Generative Prior for Image Restoration

BIGPrior: Towards Decoupling Learned Prior Hallucination and Data Fidelity in Image Restoration Authors: Majed El Helou, and Sabine Süsstrunk {Note: p

PClean: A Domain-Specific Probabilistic Programming Language for Bayesian Data Cleaning

PClean: A Domain-Specific Probabilistic Programming Language for Bayesian Data Cleaning Warning: This is a rapidly evolving research prototype.

Bayesian Image Reconstruction using Deep Generative Models
Bayesian Image Reconstruction using Deep Generative Models

Bayesian Image Reconstruction using Deep Generative Models R. Marinescu, D. Moyer, P. Golland For technical inquiries, please create a Github issue. F

Few-shot Relation Extraction via Bayesian Meta-learning on Relation Graphs

Few-shot Relation Extraction via Bayesian Meta-learning on Relation Graphs This is an implemetation of the paper Few-shot Relation Extraction via Baye

Supporting code for the paper
Supporting code for the paper "Dangers of Bayesian Model Averaging under Covariate Shift"

Dangers of Bayesian Model Averaging under Covariate Shift This repository contains the code to reproduce the experiments in the paper Dangers of Bayes

Comments
  • Exception in example NB

    Exception in example NB

    I'm up-to-date on pymc3 and gelato.

    ---------------------------------------------------------------------------
    AttributeError                            Traceback (most recent call last)
    /Users/twiecki/anaconda/lib/python3.6/site-packages/theano/gof/op.py in __call__(self, *inputs, **kwargs)
        624                 try:
    --> 625                     storage_map[ins] = [self._get_test_value(ins)]
        626                     compute_map[ins] = [True]
    
    /Users/twiecki/anaconda/lib/python3.6/site-packages/theano/gof/op.py in _get_test_value(cls, v)
        580         detailed_err_msg = utils.get_variable_trace_string(v)
    --> 581         raise AttributeError('%s has no test value %s' % (v, detailed_err_msg))
        582 
    
    AttributeError: Softmax.0 has no test value  
    Backtrace when that variable is created:
    
      File "/Users/twiecki/anaconda/lib/python3.6/site-packages/ipykernel/zmqshell.py", line 533, in run_cell
        return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
      File "/Users/twiecki/anaconda/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2717, in run_cell
        interactivity=interactivity, compiler=compiler, result=result)
      File "/Users/twiecki/anaconda/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2821, in run_ast_nodes
        if self.run_code(code, result):
      File "/Users/twiecki/anaconda/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2881, in run_code
        exec(code_obj, self.user_global_ns, self.user_ns)
      File "<ipython-input-18-7dd01309b711>", line 37, in <module>
        prediction = gelato.layers.get_output(network)
      File "/Users/twiecki/anaconda/lib/python3.6/site-packages/lasagne/layers/helper.py", line 190, in get_output
        all_outputs[layer] = layer.get_output_for(layer_inputs, **kwargs)
      File "/Users/twiecki/anaconda/lib/python3.6/site-packages/lasagne/layers/dense.py", line 124, in get_output_for
        return self.nonlinearity(activation)
      File "/Users/twiecki/anaconda/lib/python3.6/site-packages/lasagne/nonlinearities.py", line 44, in softmax
        return theano.tensor.nnet.softmax(x)
    
    
    During handling of the above exception, another exception occurred:
    
    ValueError                                Traceback (most recent call last)
    <ipython-input-18-7dd01309b711> in <module>()
         44                    prediction,
         45                    observed=target_var,
    ---> 46                    total_size=total_size)
    
    /Users/twiecki/working/projects/pymc/pymc3/distributions/distribution.py in __new__(cls, name, *args, **kwargs)
         35                 raise TypeError("observed needs to be data but got: {}".format(type(data)))
         36             total_size = kwargs.pop('total_size', None)
    ---> 37             dist = cls.dist(*args, **kwargs)
         38             return model.Var(name, dist, data, total_size)
         39         else:
    
    /Users/twiecki/working/projects/pymc/pymc3/distributions/distribution.py in dist(cls, *args, **kwargs)
         46     def dist(cls, *args, **kwargs):
         47         dist = object.__new__(cls)
    ---> 48         dist.__init__(*args, **kwargs)
         49         return dist
         50 
    
    /Users/twiecki/working/projects/pymc/pymc3/distributions/discrete.py in __init__(self, p, *args, **kwargs)
        429         super(Categorical, self).__init__(*args, **kwargs)
        430         try:
    --> 431             self.k = tt.shape(p)[-1].tag.test_value
        432         except AttributeError:
        433             self.k = tt.shape(p)[-1]
    
    /Users/twiecki/anaconda/lib/python3.6/site-packages/theano/gof/op.py in __call__(self, *inputs, **kwargs)
        637                         raise ValueError(
        638                             'Cannot compute test value: input %i (%s) of Op %s missing default value. %s' %
    --> 639                             (i, ins, node, detailed_err_msg))
        640                     elif config.compute_test_value == 'ignore':
        641                         # silently skip test
    
    ValueError: Cannot compute test value: input 0 (Softmax.0) of Op Shape(Softmax.0) missing default value.  
    Backtrace when that variable is created:
    
      File "/Users/twiecki/anaconda/lib/python3.6/site-packages/ipykernel/zmqshell.py", line 533, in run_cell
        return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
      File "/Users/twiecki/anaconda/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2717, in run_cell
        interactivity=interactivity, compiler=compiler, result=result)
      File "/Users/twiecki/anaconda/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2821, in run_ast_nodes
        if self.run_code(code, result):
      File "/Users/twiecki/anaconda/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2881, in run_code
        exec(code_obj, self.user_global_ns, self.user_ns)
      File "<ipython-input-18-7dd01309b711>", line 37, in <module>
        prediction = gelato.layers.get_output(network)
      File "/Users/twiecki/anaconda/lib/python3.6/site-packages/lasagne/layers/helper.py", line 190, in get_output
        all_outputs[layer] = layer.get_output_for(layer_inputs, **kwargs)
      File "/Users/twiecki/anaconda/lib/python3.6/site-packages/lasagne/layers/dense.py", line 124, in get_output_for
        return self.nonlinearity(activation)
      File "/Users/twiecki/anaconda/lib/python3.6/site-packages/lasagne/nonlinearities.py", line 44, in softmax
        return theano.tensor.nnet.softmax(x)
    
    opened by twiecki 12
  • Integrate opvi

    Integrate opvi

    I'm currently integrating recent changes in PyMC3 to gelato. There are a lot of changes. Everyone is welcome for discussion.

    Here are the most remarkable features:

    • no more with context when using gelato layers
    from gelato.layers import *
    import pymc3 as pm
    # get data somehow
    inp = InputLayer(shape)
    out = DenseLayer(inp, 1, W=NormalSpec(sd=LognormalSpec(sd=.1)))
    out = DenseLayer(out, 1, W=NormalSpec(sd=LognormalSpec(sd=.1)))
    with out.root:
        pm.Normal('y', mu=get_output(out, {inp:x}),
                  observed=y)
        approx = pm.fit(10000)
    
    • Flexible Specs you can do almost everything. What to do if we want different shapes there is an open question
    from gelato import *
    import theano.tensor as tt
    import pymc3 as pm
    func = as_spec_op(tt.nlinalg.matrix_power)
    expr0= func(NormalSpec() * LaplaceSpec(), 2)
    expr1 = expr0 / 100 - NormalSpec()
    with Model() as model:
        var = expr((10, 10))
        assert var.tag.test_value.shape == (10, 10)
        assert len(model.free_RVs) == 3
        fit(100)
    U = NormalSpec()
    V = UniformSpec()
    V = V / V.norm(2)
    W = U*V
    with pm.Model() as model:
        result = W((3, 2), name='weight_normalization')
    
    opened by ferrine 2
  • Fix example

    Fix example

    refere to #7. I've updated example using new pm.Minibatch API. All was running good with the following theanorc:

    [global]
    device=cpu
    floatX=float32
    mode=FAST_RUN
    optimizer_including=cudnn
    
    [lib]
    cnmem=0.95
    
    [nvcc]
    fastmath=True
    flags = -I/usr/local/cuda-8.0-cudnnv5.1/include -L/usr/local/cuda-8.0-cudnnv5.1/lib64
    
    [blas]
    ldflag = -L/usr/lib/openblas-base -Lusr/local/cuda-8.0-cudnnv5.1/lib64 -lopenblas
    
    [DebugMode]
    check_finite=1
    
    [cuda]
    root=/usr/local/cuda-8.0-cudnnv5.1/
    

    pip freeze output

    alabaster==0.7.10
    algopy==0.5.3
    Babel==2.4.0
    bleach==2.0.0
    CommonMark==0.5.4
    cycler==0.10.0
    Cython==0.25.2
    decorator==4.0.11
    docutils==0.13.1
    entrypoints==0.2.2
    -e git+https://github.com/ferrine/[email protected]#egg=gelato
    h5py==2.7.0
    html5lib==0.999999999
    imagesize==0.7.1
    ipykernel==4.6.1
    ipython==6.0.0
    ipython-genutils==0.2.0
    ipywidgets==6.0.0
    Jinja2==2.9.6
    joblib==0.11
    jsonschema==2.6.0
    jupyter==1.0.0
    jupyter-client==5.0.1
    jupyter-console==5.1.0
    jupyter-core==4.3.0
    Keras==2.0.4
    Lasagne==0.2.dev1
    Mako==1.0.6
    MarkupSafe==1.0
    matplotlib==2.0.0
    mistune==0.7.4
    more-itertools==3.1.0
    nbconvert==5.1.1
    nbformat==4.3.0
    nbsphinx==0.2.13
    nose==1.3.7
    notebook==5.0.0
    numdifftools==0.9.20
    numpy==1.13.0
    pandas==0.20.1
    pandocfilters==1.4.1
    patsy==0.4.1
    pexpect==4.2.1
    pickleshare==0.7.4
    prompt-toolkit==1.0.14
    ptyprocess==0.5.1
    Pygments==2.2.0
    pygpu==0.6.5
    -e git+https://github.com/ferrine/[email protected]#egg=pymc3
    pymongo==3.4.0
    pyparsing==2.2.0
    python-dateutil==2.6.0
    pytz==2017.2
    PyYAML==3.12
    pyzmq==16.0.2
    qtconsole==4.3.0
    recommonmark==0.4.0
    requests==2.13.0
    scikit-learn==0.18.1
    scipy==0.19.1
    seaborn==0.7.1
    simplegeneric==0.8.1
    six==1.10.0
    sklearn==0.0
    snowballstemmer==1.2.1
    Sphinx==1.5.5
    terminado==0.6
    testpath==0.3
    Theano==0.10.0.dev1
    tornado==4.5.1
    tqdm==4.11.2
    traitlets==4.3.2
    wcwidth==0.1.7
    webencodings==0.5.1
    widgetsnbextension==2.0.0
    xmltodict==0.11.0
    
    opened by ferrine 0
  • Not compatible with latest version of pymc3

    Not compatible with latest version of pymc3

    When I attempt to import gelato, it fails with the following error message:

    ---> 19 class LayerModelMeta(pm.model.InitContextMeta):
         20     """Magic comes here
         21     """
    
    AttributeError: module 'pymc3.model' has no attribute 'InitContextMeta'
    

    I believe that InitContextMeta no longer exists in pymc3; it's been merged with ContextMeta.

    I don't know if there are plans to update this repository anytime soon, although it does seem like a useful tool, so it would be great if it worked with the latest pymc3.

    opened by quevivasbien 2
Releases(v0.1.0)
Owner
Maxim Kochurov
Researcher @ NTechLab; MSU/Skoltech; Core Dev @ PyMC3, Geoopt
Maxim Kochurov
The MATH Dataset

Measuring Mathematical Problem Solving With the MATH Dataset This is the repository for Measuring Mathematical Problem Solving With the MATH Dataset b

Dan Hendrycks 267 Dec 26, 2022
A new framework, collaborative cascade prediction based on graph neural networks (CCasGNN) to jointly utilize the structural characteristics, sequence features, and user profiles.

CCasGNN A new framework, collaborative cascade prediction based on graph neural networks (CCasGNN) to jointly utilize the structural characteristics,

5 Apr 29, 2022
📚 Papermill is a tool for parameterizing, executing, and analyzing Jupyter Notebooks.

papermill is a tool for parameterizing, executing, and analyzing Jupyter Notebooks. Papermill lets you: parameterize notebooks execute notebooks This

nteract 5.1k Jan 03, 2023
The implementation of "Bootstrapping Semantic Segmentation with Regional Contrast".

ReCo - Regional Contrast This repository contains the source code of ReCo and baselines from the paper, Bootstrapping Semantic Segmentation with Regio

Shikun Liu 128 Dec 30, 2022
A unified framework for machine learning with time series

Welcome to sktime A unified framework for machine learning with time series We provide specialized time series algorithms and scikit-learn compatible

The Alan Turing Institute 6k Jan 08, 2023
Deep motion generator collections

GenMotion GenMotion (/gen’motion/) is a Python library for making skeletal animations. It enables easy dataset loading and experiment sharing for synt

23 May 24, 2022
S2s2net - Sentinel-2 Super-Resolution Segmentation Network

S2S2Net Sentinel-2 Super-Resolution Segmentation Network Getting started Install

Wei Ji 10 Nov 10, 2022
Pansharpening by convolutional neural networks in the full resolution framework

Z-PNN: Zoom Pansharpening Neural Network Pansharpening by convolutional neural networks in the full resolution framework is a deep learning method for

20 Nov 24, 2022
CFC-Net: A Critical Feature Capturing Network for Arbitrary-Oriented Object Detection in Remote Sensing Images

CFC-Net This project hosts the official implementation for the paper: CFC-Net: A Critical Feature Capturing Network for Arbitrary-Oriented Object Dete

ming71 55 Dec 12, 2022
Python with OpenCV - MediaPip Framework Hand Detection

Python HandDetection Python with OpenCV - MediaPip Framework Hand Detection Explore the docs » Contact Me About The Project It is a Computer vision pa

2 Jan 07, 2022
PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices.

PyTorch-LIT PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices. With

Amin Rezaei 157 Dec 11, 2022
Pytorch implementation for "Implicit Semantic Response Alignment for Partial Domain Adaptation"

Implicit-Semantic-Response-Alignment Pytorch implementation for "Implicit Semantic Response Alignment for Partial Domain Adaptation" Prerequisites pyt

4 Dec 19, 2022
Improving Object Detection by Label Assignment Distillation

Improving Object Detection by Label Assignment Distillation This is the official implementation of the WACV 2022 paper Improving Object Detection by L

Cybercore Co. Ltd 51 Dec 08, 2022
An Industrial Grade Federated Learning Framework

DOC | Quick Start | 中文 FATE (Federated AI Technology Enabler) is an open-source project initiated by Webank's AI Department to provide a secure comput

Federated AI Ecosystem 4.8k Jan 09, 2023
NeurIPS 2021, self-supervised 6D pose on category level

SE(3)-eSCOPE video | paper | website Leveraging SE(3) Equivariance for Self-Supervised Category-Level Object Pose Estimation Xiaolong Li, Yijia Weng,

Xiaolong 63 Nov 22, 2022
Official code of paper: MovingFashion: a Benchmark for the Video-to-Shop Challenge

SEAM Match-RCNN Official code of MovingFashion: a Benchmark for the Video-to-Shop Challenge paper Installation Requirements: Pytorch 1.5.1 or more rec

HumaticsLAB 31 Oct 10, 2022
A Collection of LiDAR-Camera-Calibration Papers, Toolboxes and Notes

A Collection of LiDAR-Camera-Calibration Papers, Toolboxes and Notes

443 Jan 06, 2023
Python scripts form performing stereo depth estimation using the CoEx model in ONNX.

ONNX-CoEx-Stereo-Depth-estimation Python scripts form performing stereo depth estimation using the CoEx model in ONNX. Stereo depth estimation on the

Ibai Gorordo 8 Dec 29, 2022
Code for our paper "Interactive Analysis of CNN Robustness"

Perturber Code for our paper "Interactive Analysis of CNN Robustness" Datasets Feature visualizations: Google Drive Fine-tuning checkpoints as saved m

Stefan Sietzen 0 Aug 17, 2021
Low-code/No-code approach for deep learning inference on devices

EzEdgeAI A concept project that uses a low-code/no-code approach to implement deep learning inference on devices. It provides a componentized framewor

On-Device AI Co., Ltd. 7 Apr 05, 2022