Bayesian dessert for Lasagne

Overview

Gelato

Coverage Status

Bayesian dessert for Lasagne

Recent results in Bayesian statistics for constructing robust neural networks have proved that it is one of the best ways to deal with uncertainty, overfitting but still having good performance. Gelato will help to use bayes for neural networks. Library heavily relies on Theano, Lasagne and PyMC3.

Installation

  • from github (assumes bleeding edge pymc3 installed)
    # pip install git+git://github.com/pymc-devs/pymc3.git
    pip install git+https://github.com/ferrine/gelato.git
  • from source
    git clone https://github.com/ferrine/gelato
    pip install -r gelato/requirements.txt
    pip install -e gelato

Usage

I use generic approach for decorating all Lasagne at once. Thus, for using Gelato you need to replace import statements for layers only. For constructing a network you need to be the in pm.Model context environment.

Warning

  • lasagne.layers.noise is not supported
  • lasagne.layers.normalization is not supported (theano problems with default updates)
  • functions from lasagne.layers are hidden in gelato as they use Lasagne classes. Some exceptions are done for lasagne.layers.helpers. I'll try to solve the problem generically in future.

Examples

For comprehensive example of using Gelato you can reference this notebook

Life Hack

Any spec class can be used standalone so feel free to use it everywhere

References

Charles Blundell et al: "Weight Uncertainty in Neural Networks" (arXiv preprint arXiv:1505.05424)

You might also like...
Bayesian optimization in PyTorch

BoTorch is a library for Bayesian Optimization built on PyTorch. BoTorch is currently in beta and under active development! Why BoTorch ? BoTorch Prov

Safe Bayesian Optimization
Safe Bayesian Optimization

SafeOpt - Safe Bayesian Optimization This code implements an adapted version of the safe, Bayesian optimization algorithm, SafeOpt [1], [2]. It also p

Bayesian Optimization using GPflow

Note: This package is for use with GPFlow 1. For Bayesian optimization using GPFlow 2 please see Trieste, a joint effort with Secondmind. GPflowOpt GP

Code for
Code for "Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations"

Infinitely Deep Bayesian Neural Networks with SDEs This library contains JAX and Pytorch implementations of neural ODEs and Bayesian layers for stocha

(under submission) Bayesian Integration of a Generative Prior for Image Restoration
(under submission) Bayesian Integration of a Generative Prior for Image Restoration

BIGPrior: Towards Decoupling Learned Prior Hallucination and Data Fidelity in Image Restoration Authors: Majed El Helou, and Sabine Süsstrunk {Note: p

PClean: A Domain-Specific Probabilistic Programming Language for Bayesian Data Cleaning

PClean: A Domain-Specific Probabilistic Programming Language for Bayesian Data Cleaning Warning: This is a rapidly evolving research prototype.

Bayesian Image Reconstruction using Deep Generative Models
Bayesian Image Reconstruction using Deep Generative Models

Bayesian Image Reconstruction using Deep Generative Models R. Marinescu, D. Moyer, P. Golland For technical inquiries, please create a Github issue. F

Few-shot Relation Extraction via Bayesian Meta-learning on Relation Graphs

Few-shot Relation Extraction via Bayesian Meta-learning on Relation Graphs This is an implemetation of the paper Few-shot Relation Extraction via Baye

Supporting code for the paper
Supporting code for the paper "Dangers of Bayesian Model Averaging under Covariate Shift"

Dangers of Bayesian Model Averaging under Covariate Shift This repository contains the code to reproduce the experiments in the paper Dangers of Bayes

Comments
  • Exception in example NB

    Exception in example NB

    I'm up-to-date on pymc3 and gelato.

    ---------------------------------------------------------------------------
    AttributeError                            Traceback (most recent call last)
    /Users/twiecki/anaconda/lib/python3.6/site-packages/theano/gof/op.py in __call__(self, *inputs, **kwargs)
        624                 try:
    --> 625                     storage_map[ins] = [self._get_test_value(ins)]
        626                     compute_map[ins] = [True]
    
    /Users/twiecki/anaconda/lib/python3.6/site-packages/theano/gof/op.py in _get_test_value(cls, v)
        580         detailed_err_msg = utils.get_variable_trace_string(v)
    --> 581         raise AttributeError('%s has no test value %s' % (v, detailed_err_msg))
        582 
    
    AttributeError: Softmax.0 has no test value  
    Backtrace when that variable is created:
    
      File "/Users/twiecki/anaconda/lib/python3.6/site-packages/ipykernel/zmqshell.py", line 533, in run_cell
        return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
      File "/Users/twiecki/anaconda/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2717, in run_cell
        interactivity=interactivity, compiler=compiler, result=result)
      File "/Users/twiecki/anaconda/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2821, in run_ast_nodes
        if self.run_code(code, result):
      File "/Users/twiecki/anaconda/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2881, in run_code
        exec(code_obj, self.user_global_ns, self.user_ns)
      File "<ipython-input-18-7dd01309b711>", line 37, in <module>
        prediction = gelato.layers.get_output(network)
      File "/Users/twiecki/anaconda/lib/python3.6/site-packages/lasagne/layers/helper.py", line 190, in get_output
        all_outputs[layer] = layer.get_output_for(layer_inputs, **kwargs)
      File "/Users/twiecki/anaconda/lib/python3.6/site-packages/lasagne/layers/dense.py", line 124, in get_output_for
        return self.nonlinearity(activation)
      File "/Users/twiecki/anaconda/lib/python3.6/site-packages/lasagne/nonlinearities.py", line 44, in softmax
        return theano.tensor.nnet.softmax(x)
    
    
    During handling of the above exception, another exception occurred:
    
    ValueError                                Traceback (most recent call last)
    <ipython-input-18-7dd01309b711> in <module>()
         44                    prediction,
         45                    observed=target_var,
    ---> 46                    total_size=total_size)
    
    /Users/twiecki/working/projects/pymc/pymc3/distributions/distribution.py in __new__(cls, name, *args, **kwargs)
         35                 raise TypeError("observed needs to be data but got: {}".format(type(data)))
         36             total_size = kwargs.pop('total_size', None)
    ---> 37             dist = cls.dist(*args, **kwargs)
         38             return model.Var(name, dist, data, total_size)
         39         else:
    
    /Users/twiecki/working/projects/pymc/pymc3/distributions/distribution.py in dist(cls, *args, **kwargs)
         46     def dist(cls, *args, **kwargs):
         47         dist = object.__new__(cls)
    ---> 48         dist.__init__(*args, **kwargs)
         49         return dist
         50 
    
    /Users/twiecki/working/projects/pymc/pymc3/distributions/discrete.py in __init__(self, p, *args, **kwargs)
        429         super(Categorical, self).__init__(*args, **kwargs)
        430         try:
    --> 431             self.k = tt.shape(p)[-1].tag.test_value
        432         except AttributeError:
        433             self.k = tt.shape(p)[-1]
    
    /Users/twiecki/anaconda/lib/python3.6/site-packages/theano/gof/op.py in __call__(self, *inputs, **kwargs)
        637                         raise ValueError(
        638                             'Cannot compute test value: input %i (%s) of Op %s missing default value. %s' %
    --> 639                             (i, ins, node, detailed_err_msg))
        640                     elif config.compute_test_value == 'ignore':
        641                         # silently skip test
    
    ValueError: Cannot compute test value: input 0 (Softmax.0) of Op Shape(Softmax.0) missing default value.  
    Backtrace when that variable is created:
    
      File "/Users/twiecki/anaconda/lib/python3.6/site-packages/ipykernel/zmqshell.py", line 533, in run_cell
        return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
      File "/Users/twiecki/anaconda/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2717, in run_cell
        interactivity=interactivity, compiler=compiler, result=result)
      File "/Users/twiecki/anaconda/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2821, in run_ast_nodes
        if self.run_code(code, result):
      File "/Users/twiecki/anaconda/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2881, in run_code
        exec(code_obj, self.user_global_ns, self.user_ns)
      File "<ipython-input-18-7dd01309b711>", line 37, in <module>
        prediction = gelato.layers.get_output(network)
      File "/Users/twiecki/anaconda/lib/python3.6/site-packages/lasagne/layers/helper.py", line 190, in get_output
        all_outputs[layer] = layer.get_output_for(layer_inputs, **kwargs)
      File "/Users/twiecki/anaconda/lib/python3.6/site-packages/lasagne/layers/dense.py", line 124, in get_output_for
        return self.nonlinearity(activation)
      File "/Users/twiecki/anaconda/lib/python3.6/site-packages/lasagne/nonlinearities.py", line 44, in softmax
        return theano.tensor.nnet.softmax(x)
    
    opened by twiecki 12
  • Integrate opvi

    Integrate opvi

    I'm currently integrating recent changes in PyMC3 to gelato. There are a lot of changes. Everyone is welcome for discussion.

    Here are the most remarkable features:

    • no more with context when using gelato layers
    from gelato.layers import *
    import pymc3 as pm
    # get data somehow
    inp = InputLayer(shape)
    out = DenseLayer(inp, 1, W=NormalSpec(sd=LognormalSpec(sd=.1)))
    out = DenseLayer(out, 1, W=NormalSpec(sd=LognormalSpec(sd=.1)))
    with out.root:
        pm.Normal('y', mu=get_output(out, {inp:x}),
                  observed=y)
        approx = pm.fit(10000)
    
    • Flexible Specs you can do almost everything. What to do if we want different shapes there is an open question
    from gelato import *
    import theano.tensor as tt
    import pymc3 as pm
    func = as_spec_op(tt.nlinalg.matrix_power)
    expr0= func(NormalSpec() * LaplaceSpec(), 2)
    expr1 = expr0 / 100 - NormalSpec()
    with Model() as model:
        var = expr((10, 10))
        assert var.tag.test_value.shape == (10, 10)
        assert len(model.free_RVs) == 3
        fit(100)
    U = NormalSpec()
    V = UniformSpec()
    V = V / V.norm(2)
    W = U*V
    with pm.Model() as model:
        result = W((3, 2), name='weight_normalization')
    
    opened by ferrine 2
  • Fix example

    Fix example

    refere to #7. I've updated example using new pm.Minibatch API. All was running good with the following theanorc:

    [global]
    device=cpu
    floatX=float32
    mode=FAST_RUN
    optimizer_including=cudnn
    
    [lib]
    cnmem=0.95
    
    [nvcc]
    fastmath=True
    flags = -I/usr/local/cuda-8.0-cudnnv5.1/include -L/usr/local/cuda-8.0-cudnnv5.1/lib64
    
    [blas]
    ldflag = -L/usr/lib/openblas-base -Lusr/local/cuda-8.0-cudnnv5.1/lib64 -lopenblas
    
    [DebugMode]
    check_finite=1
    
    [cuda]
    root=/usr/local/cuda-8.0-cudnnv5.1/
    

    pip freeze output

    alabaster==0.7.10
    algopy==0.5.3
    Babel==2.4.0
    bleach==2.0.0
    CommonMark==0.5.4
    cycler==0.10.0
    Cython==0.25.2
    decorator==4.0.11
    docutils==0.13.1
    entrypoints==0.2.2
    -e git+https://github.com/ferrine/[email protected]#egg=gelato
    h5py==2.7.0
    html5lib==0.999999999
    imagesize==0.7.1
    ipykernel==4.6.1
    ipython==6.0.0
    ipython-genutils==0.2.0
    ipywidgets==6.0.0
    Jinja2==2.9.6
    joblib==0.11
    jsonschema==2.6.0
    jupyter==1.0.0
    jupyter-client==5.0.1
    jupyter-console==5.1.0
    jupyter-core==4.3.0
    Keras==2.0.4
    Lasagne==0.2.dev1
    Mako==1.0.6
    MarkupSafe==1.0
    matplotlib==2.0.0
    mistune==0.7.4
    more-itertools==3.1.0
    nbconvert==5.1.1
    nbformat==4.3.0
    nbsphinx==0.2.13
    nose==1.3.7
    notebook==5.0.0
    numdifftools==0.9.20
    numpy==1.13.0
    pandas==0.20.1
    pandocfilters==1.4.1
    patsy==0.4.1
    pexpect==4.2.1
    pickleshare==0.7.4
    prompt-toolkit==1.0.14
    ptyprocess==0.5.1
    Pygments==2.2.0
    pygpu==0.6.5
    -e git+https://github.com/ferrine/[email protected]#egg=pymc3
    pymongo==3.4.0
    pyparsing==2.2.0
    python-dateutil==2.6.0
    pytz==2017.2
    PyYAML==3.12
    pyzmq==16.0.2
    qtconsole==4.3.0
    recommonmark==0.4.0
    requests==2.13.0
    scikit-learn==0.18.1
    scipy==0.19.1
    seaborn==0.7.1
    simplegeneric==0.8.1
    six==1.10.0
    sklearn==0.0
    snowballstemmer==1.2.1
    Sphinx==1.5.5
    terminado==0.6
    testpath==0.3
    Theano==0.10.0.dev1
    tornado==4.5.1
    tqdm==4.11.2
    traitlets==4.3.2
    wcwidth==0.1.7
    webencodings==0.5.1
    widgetsnbextension==2.0.0
    xmltodict==0.11.0
    
    opened by ferrine 0
  • Not compatible with latest version of pymc3

    Not compatible with latest version of pymc3

    When I attempt to import gelato, it fails with the following error message:

    ---> 19 class LayerModelMeta(pm.model.InitContextMeta):
         20     """Magic comes here
         21     """
    
    AttributeError: module 'pymc3.model' has no attribute 'InitContextMeta'
    

    I believe that InitContextMeta no longer exists in pymc3; it's been merged with ContextMeta.

    I don't know if there are plans to update this repository anytime soon, although it does seem like a useful tool, so it would be great if it worked with the latest pymc3.

    opened by quevivasbien 2
Releases(v0.1.0)
Owner
Maxim Kochurov
Researcher @ NTechLab; MSU/Skoltech; Core Dev @ PyMC3, Geoopt
Maxim Kochurov
Autoencoders pretraining using clustering

Autoencoders pretraining using clustering

IITiS PAN 2 Dec 16, 2021
RoadMap and preparation material for Machine Learning and Data Science - From beginner to expert.

ML-and-DataScience-preparation This repository has the goal to create a learning and preparation roadMap for Machine Learning Engineers and Data Scien

33 Dec 29, 2022
the official code for ICRA 2021 Paper: "Multimodal Scale Consistency and Awareness for Monocular Self-Supervised Depth Estimation"

G2S This is the official code for ICRA 2021 Paper: Multimodal Scale Consistency and Awareness for Monocular Self-Supervised Depth Estimation by Hemang

NeurAI 4 Jul 27, 2022
TransZero++: Cross Attribute-guided Transformer for Zero-Shot Learning

TransZero++ This repository contains the testing code for the paper "TransZero++: Cross Attribute-guided Transformer for Zero-Shot Learning" submitted

Shiming Chen 6 Aug 16, 2022
PyTorch implementations of the NeRF model described in "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis"

PyTorch NeRF and pixelNeRF NeRF: Tiny NeRF: pixelNeRF: This repository contains minimal PyTorch implementations of the NeRF model described in "NeRF:

Michael A. Alcorn 178 Dec 20, 2022
3D Human Pose Machines with Self-supervised Learning

3D Human Pose Machines with Self-supervised Learning Keze Wang, Liang Lin, Chenhan Jiang, Chen Qian, and Pengxu Wei, “3D Human Pose Machines with Self

Chenhan Jiang 398 Dec 20, 2022
This repository is maintained for the scientific paper tittled " Study of keyword extraction techniques for Electric Double Layer Capacitor domain using text similarity indexes: An experimental analysis "

kwd-extraction-study This repository is maintained for the scientific paper tittled " Study of keyword extraction techniques for Electric Double Layer

ping 543f 1 Dec 05, 2022
YOLOv5 in PyTorch > ONNX > CoreML > TFLite

This repository represents Ultralytics open-source research into future object detection methods, and incorporates lessons learned and best practices evolved over thousands of hours of training and e

Ultralytics 34.1k Dec 31, 2022
Finite-temperature variational Monte Carlo calculation of uniform electron gas using neural canonical transformation.

CoulombGas This code implements the neural canonical transformation approach to the thermodynamic properties of uniform electron gas. Building on JAX,

FermiFlow 9 Mar 03, 2022
Cooperative Driving Dataset: a dataset for multi-agent driving scenarios

Cooperative Driving Dataset (CODD) The Cooperative Driving dataset is a synthetic dataset generated using CARLA that contains lidar data from multiple

Eduardo Henrique Arnold 124 Dec 28, 2022
A curated list of awesome game datasets, and tools to artificial intelligence in games

🎮 Awesome Game Datasets In computer science, Artificial Intelligence (AI) is intelligence demonstrated by machines. Its definition, AI research as th

Leonardo Mauro 454 Jan 03, 2023
modelvshuman is a Python library to benchmark the gap between human and machine vision

modelvshuman is a Python library to benchmark the gap between human and machine vision. Using this library, both PyTorch and TensorFlow models can be evaluated on 17 out-of-distribution datasets with

Bethge Lab 244 Jan 03, 2023
This project helps to colorize grayscale images using multiple exemplars.

Multiple Exemplar-based Deep Colorization (Pytorch Implementation) Pretrained Model [Jitendra Chautharia](IIT Jodhpur)1,3, Prerequisites Python 3.6+ N

jitendra chautharia 3 Aug 05, 2022
This is the repository for The Machine Learning Workshops, published by AI DOJO

This is the repository for The Machine Learning Workshops, published by AI DOJO. It contains all the workshop's code with supporting project files necessary to work through the code.

AI Dojo 12 May 06, 2022
DeepLearning Anomalies Detection with Bluetooth Sensor Data

Final Year Project. Constructing models to create offline anomalies detection using Travel Time Data collected from Bluetooth sensors along the route.

1 Jan 10, 2022
Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies

To make the comparison with Animatable NeRF easier on the Human3.6M dataset, we save the quantitative results at here, which also contains the results of other methods, including Neural Body, D-NeRF,

ZJU3DV 359 Jan 08, 2023
PyTorch Implementation for Deep Metric Learning Pipelines

Easily Extendable Basic Deep Metric Learning Pipeline Karsten Roth ([email 

Karsten Roth 543 Jan 04, 2023
一些经典的CTR算法的复现; LR, FM, FFM, AFM, DeepFM,xDeepFM, PNN, DCN, DCNv2, DIFM, AutoInt, FiBiNet,AFN,ONN,DIN, DIEN ... (pytorch, tf2.0)

CTR Algorithm 根据论文, 博客, 知乎等方式学习一些CTR相关的算法 理解原理并自己动手来实现一遍 pytorch & tf2.0 保持一颗学徒的心! Schedule Model pytorch tensorflow2.0 paper LR ✔️ ✔️ \ FM ✔️ ✔️ Fac

luo han 149 Dec 20, 2022
Code for "Continuous-Time Meta-Learning with Forward Mode Differentiation" (ICLR 2022)

Continuous-Time Meta-Learning with Forward Mode Differentiation ICLR 2022 (Spotlight) - Installation - Example - Citation This repository contains the

Tristan Deleu 25 Oct 20, 2022
Image-Scaling Attacks and Defenses

Image-Scaling Attacks & Defenses This repository belongs to our publication: Erwin Quiring, David Klein, Daniel Arp, Martin Johns and Konrad Rieck. Ad

Erwin Quiring 163 Nov 21, 2022