A Lightweight Hyperparameter Optimization Tool ๐Ÿš€

Overview

Lightweight Hyperparameter Optimization ๐Ÿš€

Pyversions PyPI version Code style: black Colab

The mle-hyperopt package provides a simple and intuitive API for hyperparameter optimization of your Machine Learning Experiment (MLE) pipeline. It supports real, integer & categorical search variables and single- or multi-objective optimization.

Core features include the following:

  • API Simplicity: strategy.ask(), strategy.tell() interface & space definition.
  • Strategy Diversity: Grid, random, coordinate search, SMBO & wrapping around FAIR's nevergrad.
  • Search Space Refinement based on the top performing configs via strategy.refine(top_k=10).
  • Export of configurations to execute via e.g. python train.py --config_fname config.yaml.
  • Storage & reload search logs via strategy.save( ) , strategy.load( ) .

For a quickstart check out the notebook blog ๐Ÿ“– .

The API ๐ŸŽฎ

from mle_hyperopt import RandomSearch

# Instantiate random search class
strategy = RandomSearch(real={"lrate": {"begin": 0.1,
                                        "end": 0.5,
                                        "prior": "log-uniform"}},
                        integer={"batch_size": {"begin": 32,
                                                "end": 128,
                                                "prior": "uniform"}},
                        categorical={"arch": ["mlp", "cnn"]})

# Simple ask - eval - tell API
configs = strategy.ask(5)
values = [train_network(**c) for c in configs]
strategy.tell(configs, values)

Implemented Search Types ๐Ÿ”ญ

Search Type Description search_config
drawing GridSearch Search over list of discrete values -
drawing RandomSearch Random search over variable ranges refine_after, refine_top_k
drawing CoordinateSearch Coordinate-wise optimization with fixed defaults order, defaults
drawing SMBOSearch Sequential model-based optimization base_estimator, acq_function, n_initial_points
drawing NevergradSearch Multi-objective nevergrad wrapper optimizer, budget_size, num_workers

Variable Types & Hyperparameter Spaces ๐ŸŒ

Variable Type Space Specification
drawing real Real-valued Dict: begin, end, prior/bins (grid)
drawing integer Integer-valued Dict: begin, end, prior/bins (grid)
drawing categorical Categorical List: Values to search over

Installation โณ

A PyPI installation is available via:

pip install mle-hyperopt

Alternatively, you can clone this repository and afterwards 'manually' install it:

git clone https://github.com/RobertTLange/mle-hyperopt.git
cd mle-hyperopt
pip install -e .

Further Options ๐Ÿšด

Saving & Reloading Logs ๐Ÿช

# Storing & reloading of results from .pkl
strategy.save("search_log.json")
strategy = RandomSearch(..., reload_path="search_log.json")

# Or manually add info after class instantiation
strategy = RandomSearch(...)
strategy.load("search_log.json")

Search Decorator ๐Ÿงถ

from mle_hyperopt import hyperopt

@hyperopt(strategy_type="grid",
          num_search_iters=25,
          real={"x": {"begin": 0., "end": 0.5, "bins": 5},
                "y": {"begin": 0, "end": 0.5, "bins": 5}})
def circle(config):
    distance = abs((config["x"] ** 2 + config["y"] ** 2))
    return distance

strategy = circle()

Storing Configuration Files ๐Ÿ“‘

# Store 2 proposed configurations - eval_0.yaml, eval_1.yaml
strategy.ask(2, store=True)
# Store with explicit configuration filenames - conf_0.yaml, conf_1.yaml
strategy.ask(2, store=True, config_fnames=["conf_0.yaml", "conf_1.yaml"])

Retrieving Top Performers & Visualizing Results ๐Ÿ“‰

# Get the top k best performing configurations
id, configs, values = strategy.get_best(top_k=4)

# Plot timeseries of best performing score over search iterations
strategy.plot_best()

# Print out ranking of best performers
strategy.print_ranking(top_k=3)

Refining the Search Space of Your Strategy ๐Ÿช“

# Refine the search space after 5 & 10 iterations based on top 2 configurations
strategy = RandomSearch(real={"lrate": {"begin": 0.1,
                                        "end": 0.5,
                                        "prior": "log-uniform"}},
                        integer={"batch_size": {"begin": 1,
                                                "end": 5,
                                                "prior": "uniform"}},
                        categorical={"arch": ["mlp", "cnn"]},
                        search_config={"refine_after": [5, 10],
                                       "refine_top_k": 2})

# Or do so manually using `refine` method
strategy.tell(...)
strategy.refine(top_k=2)

Note that the search space refinement is only implemented for random, SMBO and nevergrad-based search strategies.

Development & Milestones for Next Release

You can run the test suite via python -m pytest -vv tests/. If you find a bug or are missing your favourite feature, feel free to contact me @RobertTLange or create an issue ๐Ÿค— .

Comments
  • [FEATURE] Hyperband

    [FEATURE] Hyperband

    Hi! I was wondering if the Hyperband hyperparameter algorithm is something you want implemented.

    I'm willing to spend some time working on it if there's interest.

    opened by colligant 5
  • [FEATURE] Option to pickle the whole strategy

    [FEATURE] Option to pickle the whole strategy

    Right now strategy.save produces a JSON with the log. Any reason you didn't opt for (or have an option of) pickling the whole strategy? Two motivations for this:

    1. Not having to re-init the strategy with all the args/kwargs
    2. Not having to loop through tell! SMBO can take quite some time to do this.
    opened by alexander-soare 4
  • Type checking strategy.log could be made more flexible?

    Type checking strategy.log could be made more flexible?

    Yay first issue! Congrats Robert, this is a great interface. Haven't used a hyperopt library in a while and this felt so easy to pick up.


    For example https://github.com/RobertTLange/mle-hyperopt/blob/57eb806e95c854f48f8faac2b2dc182d2180d393/mle_hyperopt/search.py#L251

    raises an error if my objective is numpy.float64. Also noticed https://github.com/RobertTLange/mle-hyperopt/blob/57eb806e95c854f48f8faac2b2dc182d2180d393/mle_hyperopt/search.py#L206

    Could we just have

    isinstance(strategy.log[0]['objective'], (float, int))
    

    which would cover the numpy types?

    opened by alexander-soare 4
  • Successive Halving, Hyperband, PBT

    Successive Halving, Hyperband, PBT

    • [x] Robust type checking with isinstance(self.log[0]["objective"], (float, int, np.integer, np.float))
    • [x] Add improvement method indicating if score is better than best stored one
    • [x] Fix logging message when log is stored
    • [x] Add save option for best plot
    • [x] Make json serializer more robust for numpy data types
    • [x] Add possibility to save as .pkl file by providing filename in .save method ending with .pkl (issue #2)
    • [x] Add args, kwargs into decorator
    • [x] Adds synchronous Successive Halving (SuccessiveHalvingSearch - issue #3)
    • [x] Adds synchronous HyperBand (HyperbandSearch - issue #3)
    • [x] Adds synchronous PBT (PBTSearch - issue #4 )
    opened by RobertTLange 1
  • [Feature] Synchronous PBT

    [Feature] Synchronous PBT

    Move PBT ask/tell functionality from mle-toolbox experimental to mle-hyperopt. Is there any literature/empirical evidence for the importance of being asynchronous?

    enhancement 
    opened by RobertTLange 1
Releases(v0.0.7)
  • v0.0.7(Feb 20, 2022)

    Added

    • Log reloading helper for post-processing.

    Fixed

    • Bug fix in mle-search with imports of dependencies. Needed to append path.
    • Bug fix with cleaning nested dictionaries. Have to make sure not to delete entire sub-dictionary.
    Source code(tar.gz)
    Source code(zip)
  • v0.0.6(Feb 20, 2022)

    Added

    • Adds a command line interface for running a sequential search given a python script <script>.py containing a function main(config), a default configuration file <base>.yaml & a search configuration <search>.yaml. The main function should return a single scalar performance score. You can then start the search via:

      mle-search <script>.py --base_config <base>.yaml --search_config <search>.yaml --num_iters <search_iters>
      

      Or short via:

      mle-search <script>.py -base <base>.yaml -search <search>.yaml -iters <search_iters>
      
    • Adds doc-strings to all functionalities.

    Changed

    • Make it possible to optimize parameters in nested dictionaries. Added helpers flatten_config and unflatten_config. For shaping 'sub1/sub2/vname' <-> {sub1: {sub2: {vname: v}}}
    • Make start-up message also print fixed parameter settings.
    • Cleaned up decorator with the help of Strategies wrapper.
    Source code(tar.gz)
    Source code(zip)
  • v0.0.5(Jan 5, 2022)

    Added

    • Adds possibility to store and reload entire strategies as pkl file (as asked for in issue #2).
    • Adds improvement method indicating if score is better than best stored one
    • Adds save option for best plot
    • Adds args, kwargs into decorator
    • Adds synchronous Successive Halving (SuccessiveHalvingSearch - issue #3)
    • Adds synchronous HyperBand (HyperbandSearch - issue #3)
    • Adds synchronous PBT (PBTSearch - issue #4)
    • Adds option to save log in tell method
    • Adds small torch mlp example for SH/Hyperband/PBT w. logging/scheduler
    • Adds print welcome/update message for strategy specific info

    Changed

    • Major internal restructuring:
      • clean_data: Get rid of extra data provided in configuration file
      • tell_search: Update model of search strategy (e.g. SMBO/Nevergrad)
      • log_search: Add search specific log data to evaluation log
      • update_search: Refine search space/change active strategy etc.
    • Also allow to store checkpoint of trained models in tell method.
    • Fix logging message when log is stored
    • Make json serializer more robust for numpy data types
    • Robust type checking with isinstance(self.log[0]["objective"], (float, int, np.integer, np.float))
    • Update NB to include mle-scheduler example
    • Make PBT explore robust for integer/categorical valued hyperparams
    • Calculate total batches & their sizes for hyperband
    Source code(tar.gz)
    Source code(zip)
  • v0.0.3(Oct 24, 2021)

    • Fixes CoordinateSearch active grid search dimension updating. We have to account for the fact that previous coordinates are not evaluated again after switching the active variable.
    • Generalizes NevergradSearch to wrap around all search strategies.
    • Adds rich logging to all console print statements.
    • Updates documentation and adds text to getting_started.ipynb.
    Source code(tar.gz)
    Source code(zip)
  • v0.0.2(Oct 20, 2021)

    • Fixes import bug when using PyPi installation.
    • Enhances documentation and test coverage.
    • Adds search space refinement for nevergrad and smbo search strategies via refine_after and refine_top_k:
    strategy = SMBOSearch(
            real={"lrate": {"begin": 0.1, "end": 0.5, "prior": "uniform"}},
            integer={"batch_size": {"begin": 1, "end": 5, "prior": "uniform"}},
            categorical={"arch": ["mlp", "cnn"]},
            search_config={
                "base_estimator": "GP",
                "acq_function": "gp_hedge",
                "n_initial_points": 5,
                "refine_after": 5,
                "refine_top_k": 2,
            },
            seed_id=42,
            verbose=True
        )
    
    • Adds additional strategy boolean option maximize_objective to maximize instead of performing default black-box minimization.
    Source code(tar.gz)
    Source code(zip)
  • v0.0.1(Oct 16, 2021)

    Base API implementation:

    from mle_hyperopt import RandomSearch
    
    # Instantiate random search class
    strategy = RandomSearch(real={"lrate": {"begin": 0.1,
                                            "end": 0.5,
                                            "prior": "log-uniform"}},
                            integer={"batch_size": {"begin": 32,
                                                    "end": 128,
                                                    "prior": "uniform"}},
                            categorical={"arch": ["mlp", "cnn"]})
    
    # Simple ask - eval - tell API
    configs = strategy.ask(5)
    values = [train_network(**c) for c in configs]
    strategy.tell(configs, values)
    
    Source code(tar.gz)
    Source code(zip)
Owner
Robert Lange
Deep Something @ TU Berlin ๐Ÿ•ต๏ธ
Robert Lange
Kaggler is a Python package for lightweight online machine learning algorithms and utility functions for ETL and data analysis.

Kaggler is a Python package for lightweight online machine learning algorithms and utility functions for ETL and data analysis. It is distributed under the MIT License.

Jeong-Yoon Lee 720 Dec 25, 2022
๐Ÿ’€mummify: a version control tool for machine learning

mummify is a version control tool for machine learning. It's simple, fast, and designed for model prototyping.

Max Humber 43 Jul 09, 2022
The project's goal is to show a real world application of image segmentation using k means algorithm

The project's goal is to show a real world application of image segmentation using k means algorithm

2 Jan 22, 2022
BASTA: The BAyesian STellar Algorithm

BASTA: BAyesian STellar Algorithm Current stable version: v1.0 Important note: BASTA is developed for Python 3.8, but Python 3.7 should work as well.

BASTA team 16 Nov 15, 2022
GroundSeg Clustering Optimized Kdtree

ground seg and clustering based on kitti velodyne data, and a additional optimized kdtree for knn and radius nn search

2 Dec 02, 2021
Python library which makes it possible to dynamically mask/anonymize data using JSON string or python dict rules in a PySpark environment.

pyspark-anonymizer Python library which makes it possible to dynamically mask/anonymize data using JSON string or python dict rules in a PySpark envir

6 Jun 30, 2022
Dive into Machine Learning

Dive into Machine Learning Hi there! You might find this guide helpful if: You know Python or you're learning it ๐Ÿ You're new to Machine Learning You

Michael Floering 11.1k Jan 03, 2023
Microsoft contributing libraries, tools, recipes, sample codes and workshop contents for machine learning & deep learning.

Microsoft contributing libraries, tools, recipes, sample codes and workshop contents for machine learning & deep learning.

Microsoft 366 Jan 03, 2023
Decision tree is the most powerful and popular tool for classification and prediction

Diabetes Prediction Using Decision Tree Introduction Decision tree is the most powerful and popular tool for classification and prediction. A Decision

Arjun U 1 Jan 23, 2022
The Emergence of Individuality

The Emergence of Individuality

16 Jul 20, 2022
A linear regression model for house price prediction

Linear_Regression_Model A linear regression model for house price prediction. This code is using these packages, so please make sure your have install

ShawnWang 1 Nov 29, 2021
Simulate & classify transient absorption spectroscopy (TAS) spectral features for bulk semiconducting materials (Post-DFT)

PyTASER PyTASER is a Python (3.9+) library and set of command-line tools for classifying spectral features in bulk materials, post-DFT. The goal of th

Materials Design Group 4 Dec 27, 2022
A repository to work on Machine Learning course. Select an algorithm to classify writer's gender, of Hebrew texts.

MachineLearning A repository to work on Machine Learning course. Select an algorithm to classify writer's gender, of Hebrew texts. Tested algorithms:

Haim Adrian 1 Feb 01, 2022
CinnaMon is a Python library which offers a number of tools to detect, explain, and correct data drift in a machine learning system

CinnaMon is a Python library which offers a number of tools to detect, explain, and correct data drift in a machine learning system

Zelros 67 Dec 28, 2022
A Powerful Serverless Analysis Toolkit That Takes Trial And Error Out of Machine Learning Projects

KXY: A Seemless API to 10x The Productivity of Machine Learning Engineers Documentation https://www.kxy.ai/reference/ Installation From PyPi: pip inst

KXY Technologies, Inc. 35 Jan 02, 2023
Dragonfly is an open source python library for scalable Bayesian optimisation.

Dragonfly is an open source python library for scalable Bayesian optimisation. Bayesian optimisation is used for optimising black-box functions whose

744 Jan 02, 2023
100 Days of Machine and Deep Learning Code

๐Ÿ’ฏ Days of Machine Learning and Deep Learning Code MACHINE LEARNING TOPICS COVERED - FROM SCRATCH Linear Regression Logistic Regression K Means Cluste

Tanishq Gautam 66 Nov 02, 2022
flexible time-series processing & feature extraction

A corona statistics and information telegram bot.

PreDiCT.IDLab 206 Dec 28, 2022
NCVX (NonConVeX): A User-Friendly and Scalable Package for Nonconvex Optimization in Machine Learning.

NCVX (NonConVeX): A User-Friendly and Scalable Package for Nonconvex Optimization in Machine Learning.

SUN Group @ UMN 28 Aug 03, 2022
Coursera Machine Learning - Python code

Coursera Machine Learning This repository contains python implementations of certain exercises from the course by Andrew Ng. For a number of assignmen

Jordi Warmenhoven 859 Dec 10, 2022