A web-based application for quick, scalable, and automated hyperparameter tuning and stacked ensembling in Python.

Overview

Xcessiv

PyPI license PyPI Build Status

Xcessiv is a tool to help you create the biggest, craziest, and most excessive stacked ensembles you can think of.

Stacked ensembles are simple in theory. You combine the predictions of smaller models and feed those into another model. However, in practice, implementing them can be a major headache.

Xcessiv holds your hand through all the implementation details of creating and optimizing stacked ensembles so you're free to fully define only the things you care about.

The Xcessiv process

Define your base learners and performance metrics

define_base_learner

Keep track of hundreds of different model-hyperparameter combinations

list_base_learner

Effortlessly choose your base learners and create an ensemble with the click of a button

ensemble

Features

  • Fully define your data source, cross-validation process, relevant metrics, and base learners with Python code
  • Any model following the Scikit-learn API can be used as a base learner
  • Task queue based architecture lets you take full advantage of multiple cores and embarrassingly parallel hyperparameter searches
  • Direct integration with TPOT for automated pipeline construction
  • Automated hyperparameter search through Bayesian optimization
  • Easy management and comparison of hundreds of different model-hyperparameter combinations
  • Automatic saving of generated secondary meta-features
  • Stacked ensemble creation in a few clicks
  • Automated ensemble construction through greedy forward model selection
  • Export your stacked ensemble as a standalone Python file to support multiple levels of stacking

Installation and Documentation

You can find installation instructions and detailed documentation hosted here.

FAQ

Where does Xcessiv fit in the machine learning process?

Xcessiv fits in the model building part of the process after data preparation and feature engineering. At this point, there is no universally acknowledged way of determining which algorithm will work best for a particular dataset (see No Free Lunch Theorem), and while heuristic optimization methods do exist, things often break down into trial and error as you try to find the best model-hyperparameter combinations.

Stacking is an almost surefire method to improve performance beyond that of any single model, however, the complexity of proper implementation often makes it impractical to apply them in practice outside of Kaggle competitions. Xcessiv aims to make the construction of stacked ensembles as painless as possible and lower the barrier for entry.

I don't care about fancy stacked ensembles and what not, should I still use Xcessiv?

Absolutely! Even without the ensembling functionality, the sheer amount of utility provided by keeping track of the performance of hundreds, and even thousands of ML models and hyperparameter combinations is a huge boon.

How does Xcessiv generate meta-features for stacking?

You can choose whether to generate meta-features through cross-validation (stacked generalization) or with a holdout set (blending). You can read about these two methods and a lot more about stacked ensembles in the Kaggle Ensembling Guide. It's a great article and provides most of the inspiration for this project.

Contributing

Xcessiv is in its very early stages and needs the open-source community to guide it along.

There are many ways to contribute to Xcessiv. You could report a bug, suggest a feature, submit a pull request, improve documentation, and many more.

If you would like to contribute something, please visit our Contributor Guidelines.

Project Status

Xcessiv is currently in alpha and is unstable. Future versions are not guaranteed to be backwards-compatible with current project files.

Comments
  • Can't Use

    Can't Use

    Sorry for what is no doubt a stupid question:

    I've started Redis via redis-server. It says it's running on port 6379. Then I run xcessiv, but it takes me to a page that's not found. The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.. Any idea what I can do? I'm really eager to use Xcessiv.

    opened by xnmp 6
  • Automated ensembling techniques

    Automated ensembling techniques

    Working for a while with Xcessiv, I feel there's a need for some way to automate the selection of base learners in an ensemble. I'm unaware of existing techniques for this, so if anyone has any suggestions or could point me towards relevant literature, it would be greatly appreciated.

    enhancement 
    opened by reiinakano 5
  • Added more of the sklearn regressors to the presets

    Added more of the sklearn regressors to the presets

    Added the large majority of the more popular regressors of sklearn. I am aware that a few may be missing. Also, I tidied the code slightly and split the regressors and classifiers into two sections.

    opened by enisnazif 4
  • Memory management

    Memory management

    First of all, thanks! I find this project fascinating. My question/issue is about how do you handle the memory for multiple processes. By default Python will create a copy of the data per process. This is prohibitive for large datasets.

    How did you manage this problem?

    opened by alvarouc 3
  • Move .gitignore to project root and add Python ignores

    Move .gitignore to project root and add Python ignores

    I think the best practice for .gitignore is to have a single .gitignore file at the root of the project so I moved the .gitignore that was in xcessiv/ui (I think it was generated by create-react-scripts) to the project root and added some Python ignore lines.

    opened by menglewis 3
  • Added Leave One Out Crossvalidation to cvsetting.py

    Added Leave One Out Crossvalidation to cvsetting.py

    Added Leave One Out Cross validation as part of #15

    I'm keen to finish implementing all of the cv / metrics within sklearn, just wanted to make sure I was doing it right since this is my first pull request!

    opened by enisnazif 2
  • XGBRegressor model stuck in queued status

    XGBRegressor model stuck in queued status

    I tried to make a regression model to run on zillow data from kaggle available here https://www.kaggle.com/c/zillow-prize-1/data Here is a gist of my dataset extractor as well as setting up the XGBRegressor and an exception that was the last thing left in the console https://gist.github.com/jef5ez/a9b0650293f343682a58b0f0500f3332 I selected the shuffle split for both cross validation settings and added MSE as the learner metric. The base learner seems to verify fine on the boston housing data. After hitting finalize and selecting a single base learner a row shows up below but is stuck in the Queued status.

    python 3.5.2 xcessiv (0.2.2) xgboost (0.6a2)

    opened by jef5ez 2
  • The _BasePipeline in exported Python script should be _BaseComposition

    The _BasePipeline in exported Python script should be _BaseComposition

    Since scitkit-learn 0.19.x, the base class for Pipeline has changed to _BaseComposition. https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/pipeline.py When using the generated code for training, it raises a name-not-found error on newer versions of sklearn. At the moment, an easy workaround is to change two instances of the word manually in the generated script.

    opened by Mithrillion 1
  • Issues with TfidnVectorizer

    Issues with TfidnVectorizer

    Hey, great tool.

    I have a problem though when I am trying to use a TfidfVectorizer for Text Classification. When I create a Single Base Learner I get the error:

    ValueError: all the input array dimensions except for the concatenation axis must match exactly .

    The type of the X variable is an numpy.ndarray, but if I don't convert the variable X to an array then I get the error message:

    TypeError: Singleton array array(<92820x194 sparse matrix of type '<class 'numpy.float64'>' with 92820 stored elements in Compressed Sparse Row format>, dtype=object) cannot be considered a valid collection.

    I choose the preset learner setting scikit-learn Random Forest as a Base Learner Type.

    import os
    import numpy as np
    import pandas as pd
    import pickle
    from sklearn.feature_extraction.text import TfidfVectorizer
    from sklearn.ensemble import RandomForestClassifier
    
    def extract_main_dataset():
        # pandas data frame with the columns Classification, FeatureVector
        # ie:
        # 0, 'This is the feature vector'
        # 1, 'This is another feature vector' 
        # 2, 'This is yet another feature vector' 
        # 1, 'This is the last feature vector example' 
        with open('feature_vector.pik', 'rb') as rf:
            feature_vector = pickle.load(rf)
    
        y = np.array(feature_vector.Classification.values)
        title_rf_vectorizer = TfidfVectorizer(ngram_range=(2, 9),
                                              sublinear_tf=True,
                                              use_idf=True,
                                              strip_accents='ascii')
    
        title_rf_classifier = RandomForestClassifier(n_estimators=100, n_jobs=8)
        X = title_rf_vectorizer.fit_transform(feature_vector["Classification"]).toarray()
        return X, y
    
    opened by bbowler86 1
  • Valid values for metric to optimise in bayesian optimisation?

    Valid values for metric to optimise in bayesian optimisation?

    Is there a list of valid metric_to_optimise for Bayesian Optimisation?

    I am using sklearn mean_squared_regression for my base learning but when I enter that into the Bayesian Optimisation menu under metric_to_optimise I get:

    assert module.metric_to_optimize in automated_run.base_learner_origin.metric_generators
    AssertionError
    
    question 
    opened by Data-drone 1
  • 'dict_keys' object does not support indexing

    'dict_keys' object does not support indexing

    On lines 306 and 309 of views.py, trying to index a dictionary keys object will fail on Python 3 and result in a server error. The fix is simple: change all occurrences of

    base_learner_origin.validation_results.keys()[0]

    to

    list(base_learner_origin.validation_results.keys())[0]

    opened by KhaledSharif 1
  • redis.exceptions.DataError at xcessiv launch

    redis.exceptions.DataError at xcessiv launch

    Hello, When I try to launch xcessiv I get an error:

    Traceback (most recent call last): File "/PATH_TO/anaconda3/bin/xcessiv", line 10, in <module> sys.exit(main()) File "/PATH_TO/anaconda3/lib/python3.7/site-packages/xcessiv/scripts/runapp.py", line 51, in main redis_conn.get(None) # will throw exception if Redis is unavailable File "/PATH_TO/anaconda3/lib/python3.7/site-packages/redis/client.py", line 1264, in get return self.execute_command('GET', name) File "/PATH_TO/anaconda3/lib/python3.7/site-packages/redis/client.py", line 774, in execute_command connection.send_command(*args) File "/PATH_TO/anaconda3/lib/python3.7/site-packages/redis/connection.py", line 620, in send_command self.send_packed_command(self.pack_command(*args)) File "/PATH_TO/anaconda3/lib/python3.7/site-packages/redis/connection.py", line 663, in pack_command for arg in imap(self.encoder.encode, args): File "/PATH_TO/anaconda3/lib/python3.7/site-packages/redis/connection.py", line 125, in encode "byte, string or number first." % typename) redis.exceptions.DataError: Invalid input of type: 'NoneType'. Convert to a byte, string or number first.

    Previously I had to change from gevent.wsgi import WSGIServer to from gevent.pywsgi import WSGIServer as indicated in this issue

    My server is responding when I do redis-cli ping

    I am on Ubuntu 18.04, with python 3.7.3 and redis 5.0.5

    Do you have an idea to fix this? Thanks!

    opened by AlexCoul 0
  • How to import homemade modules in Xcessiv?

    How to import homemade modules in Xcessiv?

    I'm trying to import homemade module named preprocessing_115v (filename preprocessing_115v.py) into the main data extraction source code but I can't seem to find it :

    ############# import preprocessing_115v <-- where do I store the preprocessing_115v.py file for it to load here? def extract_main_dataset(): import pandas as pd df=pd.read_csv('./data.csv', sep=',',header=None) X=df.values labels=pd.read_csv('./labelsnum.csv', sep=',',header=None) y=labels.values y=y[:,0] return X, y ##############

    Amazing program by the way :-)

    opened by fcoppey 0
  • xcessiv server

    xcessiv server

    Hi This project looks very cool, but I am having some problems with the setup. I am running this in a container (my own), and I can't get the server to show up. From inside the container I can see the server running - ps shows xcessiv running and curl localhost:1994 gives me some HTML from xcessiv. From outside the container, however, there's nothing.

    I suppose that's down to the server.py file which I have now changed to this:

      1 from __future__ import absolute_import, print_function, division, unicode_literals¬                                                                              
      2 from gevent.wsgi import WSGIServer¬
      3 # import webbrowser¬
      4 ¬
      5 ¬
      6 def launch(app):¬
      7     http_server = WSGIServer(('0.0.0.0', app.config['XCESSIV_PORT']), app)¬
      8     # webbrowser.open_new('http://localhost:' + str(app.config['XCESSIV_PORT']))¬
      9     http_server.serve_forever()¬
    

    I have changed the WSGIServer setup to be open to outside connection (I suppose that's what I changed), but it's still not showing up.

    Feedback appreciated. I'd like to try this out. Thanks!

    opened by benman1 0
  • Feature Request - Backup .db file

    Feature Request - Backup .db file

    I got an error something to the effect of "Error with JSON "N" at position 8345", presumably caused by my manually editing the code for one of the base learners. Once I got this error however, none of the base learners in my project would load. I resolved it by manually deleting the base learner I had been editing from the .db file. I'll post the specifics if I can recreate it, but I'm wondering if it might be prudent to have some kind of db backup/"Last Known Good Configuration"?

    opened by Tahlor 0
  • Fix issue #63 no module named wsgi

    Fix issue #63 no module named wsgi

    In file server.py

    from gevent.wsgi import WSGIServer
    

    Has to be changed to:

    from gevent.pywsgi import WSGIServer
    

    http://www.gevent.org/api/gevent.pywsgi.html

    opened by KhaledTo 0
  • ImportError: No module named wsgi

    ImportError: No module named wsgi

    File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/xcessiv/server.py", line 2, in from gevent.wsgi import WSGIServer ImportError: No module named wsgi

    opened by xialeizhou 2
Releases(v0.5.1)
Owner
Reiichiro Nakano
I like working on awesome things with awesome people!
Reiichiro Nakano
Python Single Object Tracking Evaluation

pysot-toolkit The purpose of this repo is to provide evaluation API of Current Single Object Tracking Dataset, including VOT2016 VOT2018 VOT2018-LT OT

348 Dec 22, 2022
SpeechNAS Better Trade off between Latency and Accuracy for Large Scale Speaker Verification

SpeechNAS Better Trade off between Latency and Accuracy for Large Scale Speaker Verification

Wentao Zhu 24 May 20, 2022
Fast and exact ILP-based solvers for the Minimum Flow Decomposition (MFD) problem, and variants of it.

MFD-ILP Fast and exact ILP-based solvers for the Minimum Flow Decomposition (MFD) problem, and variants of it. The solvers are implemented using Pytho

Algorithmic Bioinformatics Group @ University of Helsinki 4 Oct 23, 2022
(Arxiv 2021) NeRF--: Neural Radiance Fields Without Known Camera Parameters

NeRF--: Neural Radiance Fields Without Known Camera Parameters Project Page | Arxiv | Colab Notebook | Data Zirui Wang¹, Shangzhe Wu², Weidi Xie², Min

Active Vision Laboratory 411 Dec 26, 2022
Code and data for ACL2021 paper Cross-Lingual Abstractive Summarization with Limited Parallel Resources.

Multi-Task Framework for Cross-Lingual Abstractive Summarization (MCLAS) The code for ACL2021 paper Cross-Lingual Abstractive Summarization with Limit

Yu Bai 43 Nov 07, 2022
Augmentation for Single-Image-Super-Resolution

SRAugmentation Augmentation for Single-Image-Super-Resolution Implimentation CutBlur Cutout CutMix Cutup CutMixup Blend RGBPermutation Identity OneOf

Yubo 6 Jun 27, 2022
Repository for Driving Style Recognition algorithms for Autonomous Vehicles

Driving Style Recognition Using Interval Type-2 Fuzzy Inference System and Multiple Experts Decision Making Created by Iago Pachêco Gomes at USP - ICM

Iago Gomes 9 Nov 28, 2022
The Official Implementation of the ICCV-2021 Paper: Semantically Coherent Out-of-Distribution Detection.

SCOOD-UDG (ICCV 2021) This repository is the official implementation of the paper: Semantically Coherent Out-of-Distribution Detection Jingkang Yang,

Jake YANG 62 Nov 21, 2022
Supplementary code for SIGGRAPH 2021 paper: Discovering Diverse Athletic Jumping Strategies

SIGGRAPH 2021: Discovering Diverse Athletic Jumping Strategies project page paper demo video Prerequisites Important Notes We suspect there are bugs i

54 Dec 06, 2022
Whisper is a file-based time-series database format for Graphite.

Whisper Overview Whisper is one of three components within the Graphite project: Graphite-Web, a Django-based web application that renders graphs and

Graphite Project 1.2k Dec 25, 2022
Generating synthetic mobility data for a realistic population with RNNs to improve utility and privacy

lbs-data Motivation Location data is collected from the public by private firms via mobile devices. Can this data also be used to serve the public goo

Alex 11 Sep 22, 2022
Image-to-image translation with conditional adversarial nets

pix2pix Project | Arxiv | PyTorch Torch implementation for learning a mapping from input images to output images, for example: Image-to-Image Translat

Phillip Isola 9.3k Jan 08, 2023
gtfs2vec - Learning GTFS Embeddings for comparing PublicTransport Offer in Microregions

gtfs2vec This is a companion repository for a gtfs2vec - Learning GTFS Embeddings for comparing PublicTransport Offer in Microregions publication. Vis

Politechnika Wrocławska - repozytorium dla informatyków 5 Oct 10, 2022
PyTorch reimplementation of the paper Involution: Inverting the Inherence of Convolution for Visual Recognition [CVPR 2021].

Involution: Inverting the Inherence of Convolution for Visual Recognition Unofficial PyTorch reimplementation of the paper Involution: Inverting the I

Christoph Reich 100 Dec 01, 2022
This is an official pytorch implementation of Lite-HRNet: A Lightweight High-Resolution Network.

Lite-HRNet: A Lightweight High-Resolution Network Introduction This is an official pytorch implementation of Lite-HRNet: A Lightweight High-Resolution

HRNet 675 Dec 25, 2022
Open source implementation of AceNAS: Learning to Rank Ace Neural Architectures with Weak Supervision of Weight Sharing

AceNAS This repo is the experiment code of AceNAS, and is not considered as an official release. We are working on integrating AceNAS as a built-in st

Yuge Zhang 6 Sep 07, 2022
Collects many various multi-modal transformer architectures, including image transformer, video transformer, image-language transformer, video-language transformer and related datasets

The repository collects many various multi-modal transformer architectures, including image transformer, video transformer, image-language transformer, video-language transformer and related datasets

Jun Chen 139 Dec 21, 2022
Code/data of the paper "Hand-Object Contact Prediction via Motion-Based Pseudo-Labeling and Guided Progressive Label Correction" (BMVC2021)

Hand-Object Contact Prediction (BMVC2021) This repository contains the code and data for the paper "Hand-Object Contact Prediction via Motion-Based Ps

Takuma Yagi 13 Nov 07, 2022
Mixed Transformer UNet for Medical Image Segmentation

MT-UNet Update 2021/11/19 Thank you for your interest in our work. We have uploaded the code of our MTUNet to help peers conduct further research on i

dotman 92 Dec 25, 2022
This is an early in-development version of training CLIP models with hivemind.

A transformer that does not hog your GPU memory This is an early in-development codebase: if you want a stable and documented hivemind codebase, look

<a href=[email protected]"> 4 Nov 06, 2022