On Anytime Learning At Macroscale

Learning from sequential data dumps

(key) Requirements

Python 3.7
Pytorch 1.9.0
Hydra 1.1.0 (pip install hydra-core & pip install hydra-submitit-launcher)

Structure

├── crlapi           
  ├── benchmark.py    # Creates the data stream, feeds it to the model and evaluates it
  ├── core.py         # Abstract classes for 
  ├── logger.py   
  ├── sl
    ├── architectures
      ├── ...         # NN architectures used in this project
    ├── clmodels
      ├── ...         # Models (e.g. Single, gEns, ..., )
    ├── streams
      ├── ...         # CIFAR and MNIST stream implementatins

Running Experiments

To run experiments, you need to call the dataset specific run file, and you need to pass the configuration of the run. We have place the configurations in the previous directory (../configs). The config structure is as follows

    ├── configs
        ├── mnist
           ├── run.py                 # run file
           ├── test_usage_gmoe.yaml   # This is the "gMoE" model
           ├── test_finetune_mlp.yaml # This is the "Single Model"
           ... 
        ├── cifar
           ├── run.py                 # run file
           ├── test_finetune_vgg.yaml # This is the "Single Model"
           ├── test_usage_gmoe.yaml   # This is the "gMoE" model
           ...

To run an e.g. mnist gMoE run, the command is (launched from the directory just above (so cd ..)

PYTHONPATH=./ python configs/mnist/run.py -cn test_usage_gmoe n_megabatches=2 replay=1 clmodel.max_epochs=200

Important arguments

n_megabatches : controls the number of megabatches. So n_megabatches=1 is your regular full dataset training
replay : whether to use replay or not
clmodel.init_from_scratch : whether to reinitialize the model at every MB. Should only be used when replay=1
device : use cuda or cpu depending on your hardware

License

alma is released under the MIT license. See LICENSE for additional details about it. See also our Terms of Use and Privacy Policy.

Anytime Learning At Macroscale

Related tags

Overview

On Anytime Learning At Macroscale

(key) Requirements

Structure

Running Experiments

Important arguments

License

Owner

Meta Research

100 Days of Machine and Deep Learning Code

Turning images into '9-pan' palettes using KMeans clustering from sklearn.

Dual Adaptive Sampling for Machine Learning Interatomic potential.

Markov bot - A Writing bot based on Markov Chain for Data Structure Lab

AP1 Transcription Factor Binding Site Prediction

This is a curated list of medical data for machine learning

A collection of neat and practical data science and machine learning projects

Diabetes Prediction with Logistic Regression

Reggy - Regressions with arbitrarily complex regularization terms

ZenML 🙏: MLOps framework to create reproducible ML pipelines for production machine learning.

A quick reference guide to the most commonly used patterns and functions in PySpark SQL

Deepchecks is a Python package for comprehensively validating your machine learning models and data with minimal effort

A Python toolkit for rule-based/unsupervised anomaly detection in time series

TensorFlow implementation of an arbitrary order Factorization Machine

A benchmark of data-centric tasks from across the machine learning lifecycle.

jaxfg - Factor graph-based nonlinear optimization library for JAX.

DistML is a Ray extension library to support large-scale distributed ML training on heterogeneous multi-node multi-GPU clusters

A complete guide to start and improve in machine learning (ML)

MegFlow - Efficient ML solutions for long-tailed demands.

Module for statistical learning, with a particular emphasis on time-dependent modelling