The Fuzzy Labs guide to the universe of open source MLOps

Last update: Dec 29, 2022

Overview

Open Source MLOps

This is the Fuzzy Labs guide to the universe of free and open source MLOps tools.

What is MLOps, anyway?
Data version control
Experiment tracking
Model training
Feature stores
Model deployment and serving
Model monitoring
Full stacks
More resources

What is MLOps anyway?

MLOps (machine learning operations) is a discipline that helps people to train, deploy and run machine learning models successfully in production environments. Because this is a new and rapidly-evolving field, there are a lot of tools out there, and new ones appear all the time. If we've missed any, then please do raise a pull request!

Data version control

Just like code, data grows and evolves over time. Data versioning tools help you to keep track of these changes.

You might wonder why you can't just store data in Git (or equivalent). There are a few reasons this doesn't work, but the main one is size: Git is designed for small text files, and typical datasets used in machine learning are just too big. Some tools, like DVC, store the data externally, but also integrate with Git so that data versions can be linked to code versions.

DVC - one of the most popular general-purpose data versioning tools.
Delta Lake - data versioning for data warehouses.
LakeFS - Transform your object storage into a Git-like repository.
Git LFS - while this doesn't specialise in machine learning use-cases, it's another popular way to version datasets.

Experiment tracking

Machine learning involves a lot of experimentation. We end up training a lot of models, most of which are never intended to go into production, but represent progressive steps towards having something production-worthy. Experiment tracking tools are there to help us keep track of each experiment. What exactly do we need to track? typically this includes the code version, data version, input parameters, training performance metrics, as well as the final model assets.

Model training

Feature stores

Feast

Model deployment and serving

Model serving is the process of taking a trained model and presenting it behind a REST API, and this enables other software components to interact with a model. To make deployment of these model servers as simple as possible, it's commonplace to run them inside Docker containers and deploy them to a container orchestration system such as Kubernetes.

Model monitoring

Full stacks

More resources

Here are some more resources for MLOps, both open-source and proprietary.

Top 10 Open Source MLOps Tools
Awesome MLOps - a mixture of open source and proprietory tools and platforms.
Best open source MLOps tools

Comments

Data Catalogues

Just a short snippet on Data Catalogues. Almost started trying to build out a custom one because I didn't know they existed. Therefore feel it's important to make some noise about them.

Open to feedback, can add more detail once I know if you guys think they fit the MLOps category. Only didn't put them under the Data Governance bin because those that focus on Data Discovery don't have so much to do with it. Happy to shuffle the content over though.

opened by GeorgePearse 3
Add feathr (feature store by Linkedin)

This PR adds Feathr

About Feathr:

Feathr is an open source enterprise-grade, high performance feature store, hosted in incubation in the LF AI & Data Foundation.

@archena Please let me know if any changes are needed for the description.

opened by SangamSwadiK 1
add new section for Model validation, with trubrics

Hello, we have recently launched Trubrics and I'd love to add it to your great list!

Trubrics helps Data Scientists validate their models, by providing them with a framework to write validations. Validations can be built purely with data science knowledge, or with feedback collected from business users on ML models.

Thanks :grinning:

opened by jeffkayne 1
Added MLEM and put description for CML

Hello! We have a new MLOPs tool we'd love to add to the awesome-open-mlops!

You can find the repo here: https://github.com/iterative/mlem Read the blog post: https://iterative.ai/blog/MLEM-release Watch the video: https://youtu.be/7h0fiZNwCnA

Let me know if you need anything else or would like to collaborate in some way! Best regards...

opened by mertbozkir 1
Added few notes on YDNBB, plus two more MIT-licensed repos

Added few hopefully interesting repos, all OS focused. I used monitoring for RecList even if (interestingly) is probably in his own category (with CheckList for example): "model testing", or something like that!

opened by jacopotagliabue 1
Update the open source MLOps repo with data annotation

Picked out the most interesting tools from this repo built by ZenML. Added a new section to our repo for data annotation. Include a link to ZenML’s repo as well.

opened by osw282 1
chore: Add envd

I'd like to share envd with the community!

envd is a machine learning development environment for data science and AI/ML engineering teams.

🐍 No Docker, only Python - Focus on writing Python code, we will take care of Docker and development environment setup.

🖨️ Built-in Jupyter/VSCode - First-class support for Jupyter and VSCode remote extension.

⏱️ Save time - Better cache management to save your time, keep the focus on the model, instead of dependencies.

☁️ Local & cloud - envd integrates seamlessly with Docker so that you can easily share, version, and publish envd environments with Docker Hub or any other OCI image registries.

🔁 Repeatable builds & reproducible results - You can reproduce the same dev environment on your laptop, public cloud VMs, or Docker containers, without any change in setup.

Signed-off-by: Ce Gao [email protected]

opened by gaocegege 1
Adds a section for model registries

👋 Hello!

This PR adds a new section for model registries and I've added both ML Flow and modelstore to it. Disclosure: I'm the author of the latter.

When reading through the other sections, I saw that this might overlap slightly with experiment tracking, which is described as:

What exactly do we need to track? typically this includes the code version, data version, input parameters, training performance metrics, as well as the final model assets.

Happy to change this around as you see fit.

Thank you for considering this contribution 🙏

opened by nlathia 1
Adds Hamilton and feature engineering section

Hamilton was created to help wrangle a feature engineering code base. It forces decoupling of feature transform logic from materialization, and results in code that is always unit testable, reusable, and documentation friendly.

I didn't see an appropriate section to add it, so I created a feature engineering section - putting it in feature stores wouldn't be the right place.

opened by skrawcz 1
Image Analysis Tools

I've synced the forks. I have no idea why my previous data catalogue addition is listed again as a new change? Will correct that if anyone can tell me where I've gone wrong.

I've hit my limit of GitHub lists so will be making greater use of this repo. Adding in image analysis tools (fiftyone - the dominant player, and dendromap - looks very powerful).

Will add links, and try to get round to adding descriptions for data catalogues soon

opened by GeorgePearse 1
Deepchecks library
Deepchecks is open source tool for testing and validating machine learning models and data. The 3 checks supported in different phases of ML pipeline are

Data Integrity (between ingestion and preprocessing step)

Train-Test Validation (Distribution and Methodology Checks) (between preprocessing and training step)

Model Performance Evaluation (evaluation step)
opened by dudeperf3ct 0

Releases(v0.1.0-alpha)

v0.1.0-alpha(Dec 13, 2021)
Initial release of the Awesome Open Source MLOps list.

Data version control

Experiment tracking

Model training

Feature stores

Model deployment and serving

Model monitoring

Full stacks

Source code(tar.gz)
Source code(zip)

Owner

Fuzzy Labs

MLOps done right

GitHub Repository

Evaluate on three different ML model for feature selection using Breast cancer data.

Anomaly-detection-Feature-Selection Evaluate on three different ML model for feature selection using Breast cancer data. ML models: SVM, KNN and MLP.

1 Mar 17, 2022

Mixing up the Invariant Information clustering architecture, with self supervised concepts from SimCLR and MoCo approaches

Self Supervised clusterer Combined IIC, and Moco architectures, with some SimCLR notions, to get state of the art unsupervised clustering while retain

9 Feb 13, 2022

ML Optimizers from scratch using JAX

Toy implementations of some popular ML optimizers using Python/JAX

38 Jul 29, 2022

A fast, distributed, high performance gradient boosting (GBT, GBDT, GBRT, GBM or MART) framework based on decision tree algorithms, used for ranking, classification and many other machine learning tasks.

Light Gradient Boosting Machine LightGBM is a gradient boosting framework that uses tree based learning algorithms. It is designed to be distributed a

14.5k Jan 07, 2023

The unified machine learning framework, enabling framework-agnostic functions, layers and libraries.

The unified machine learning framework, enabling framework-agnostic functions, layers and libraries. Contents Overview In a Nutshell Where Next? Overv

8.2k Dec 31, 2022

pywFM is a Python wrapper for Steffen Rendle's factorization machines library libFM

pywFM pywFM is a Python wrapper for Steffen Rendle's libFM. libFM is a Factorization Machine library: Factorization machines (FM) are a generic approa

251 Sep 23, 2022

This is a curated list of medical data for machine learning

Medical Data for Machine Learning This is a curated list of medical data for machine learning. This list is provided for informational purposes only,

5.4k Dec 26, 2022

Tutorials, examples, collections, and everything else that falls into the categories: pattern classification, machine learning, and data mining

**Tutorials, examples, collections, and everything else that falls into the categories: pattern classification, machine learning, and data mining.** S

4k Dec 30, 2022

AP1 Transcription Factor Binding Site Prediction

A machine learning project that predicted binding sites of AP1 transcription factor, using ChIP-Seq data and local DNA shape information.

1 Jan 21, 2022

Predict the income for each percentile of the population (Python) - FRENCH

05.income-prediction Predict the income for each percentile of the population (Python) - FRENCH Effectuez une prédiction de revenus Prérequis Pour ce

1 Feb 13, 2022

🚪✊Knock Knock: Get notified when your training ends with only two additional lines of code

Knock Knock A small library to get a notification when your training is complete or when it crashes during the process with two additional lines of co

2.5k Jan 07, 2023

MBTR is a python package for multivariate boosted tree regressors trained in parameter space.

61 Dec 19, 2022

Machine learning model evaluation made easy: plots, tables, HTML reports, experiment tracking and Jupyter notebook analysis.

sklearn-evaluation Machine learning model evaluation made easy: plots, tables, HTML reports, experiment tracking, and Jupyter notebook analysis. Suppo

354 Dec 31, 2022