Hello Simon,

I tried to compile a LightGBM model using both LLeaves and TreeLite.

I found that the compiled .so file by LLeaves is ~80% larger than the files compiled by TreeLite.

I want ask that is it possible to reduce the size of the compiled .so file by LLeaves?

Hi, We installed lleaves using pip install lleaves. We found that the prediction can only utilize one CPU core, though we set n_jobs=8, while we have 8 CPU cores. This is inconsistent with the lleaves code here: https://github.com/siboehm/lleaves/blob/master/lleaves/lleaves.py#L140

Why will that happen? Any suggestions are highly appreciated.

In lightgbm model file, pandas_categorical may be null. When call data_processing._dataframe_to_ndarray, exception happend caused by if len(cat_cols) != len(pd_traintime_categories) TypeError: object of type 'NoneType' has no len()

First of all, thank you for your impressive work. I wanted to ask if there is a way to store the compiled models in a persistent way. In my case the predictor is composed by ~100 LightGBM models, so that the compiling of the full predictor is highly time consuming. When I tried to pickle the compiled lleaves model, I got:

ValueError: ctypes objects containing pointers cannot be pickled

for which I guess there is no easy workaround. Do you know if it possible to avoid re-compilation of the original LightGBM instances? Thank you

I tried benchmarking lleaves vs treelite and found that lleaves is slightly slower than treelite. I might be doing something wrong?

I benchmark in google benchmark with batch size 1 and random features. I have ~600 trees with 450 leaves and max depth 13. Treelite is compiled with clang 10.0. I think we did see that treelite was a lot slower using GCC.

I noticed that the compile step for lleaves took severeal hours, so maybe the forest I'm using is somehow off?

In any case I think your library looks very nice :)

Xeon E2278G

------------------------------------------------------
Benchmark            Time             CPU   Iterations
------------------------------------------------------
BM_LLEAVES       32488 ns        32487 ns        21564
BM_TREELITE      27251 ns        27250 ns        25635

EPYC 7402P

------------------------------------------------------
Benchmark            Time             CPU   Iterations
------------------------------------------------------
BM_LLEAVES       38020 ns        38019 ns        18308
BM_TREELITE      32155 ns        32154 ns        21579

#include <benchmark/benchmark.h>
#include <random>
#include "lleavesheader.h"
#include "treeliteheader.h"
#include <vector>
#include <random>
#include <iostream>


constexpr int NUM_FATURES = 108;

static void BM_LLEAVES(benchmark::State& state)
{

    std::random_device dev;
    std::mt19937 rng(dev());
    std::uniform_int_distribution<std::mt19937::result_type> dist(-10,10); 

    std::size_t N = 10000000;
    std::vector<double> f;
    f.reserve(N);
    for (std::size_t i = 0; i<N; ++i){
        f.push_back(dist(rng));
    }

    double out;
    std::size_t i = 0;
    for (auto _ : state) {
        forest_root(f.data()+NUM_FATURES*i, &out, (int)0, (int)1);
        ++i;
    }
}

static void BM_TREELITE(benchmark::State& state)
{

    std::random_device dev;
    std::mt19937 rng(dev());
    std::uniform_int_distribution<std::mt19937::result_type> dist(-10,10); // distribution in range [1, 6]

    std::size_t N = 10000000;
    std::vector<Entry> f;
    f.reserve(N);
    for (std::size_t i = 0; i<N; ++i){
        auto e = DE::Entry();
        e.missing = -1;
        e.fvalue = dist(rng);
        e.qvalue = 0;
        f.push_back(e);
    }

    std::size_t i = 0;
    union DE::Entry *pFeatures = nullptr; 
    for (auto _ : state) {
        pFeatures = f.data()+NUM_FATURES*i;
        predict(pFeatures, 1);   // call treelite predict function

        ++i;
        
    }
}
BENCHMARK(BM_LLEAVES);
BENCHMARK(BM_TREELITE);
BENCHMARK_MAIN();

Hi, currently evaluating this as a potential performance enhancement on our MLOps / Inference stack.

Tought I'd give some numbers here (based on MacBook Pro 2019).

Test set up as follows: a) generate artificial data X = 1E6 x 200 float64, Y = X.sum() for regression, Y = X.sum() > 100 for binary classifier b) for n_feat in [...] -> fit model on 1000 samples and n_feat features; compile model c) for batchsize in [...] -> predict 10 times a randomly sampled batch of all data items, using (1) LGBM.predict(), (2). lleaves.predict(), (3) lleaves.predict(n_jobs=1); measure TOTAL time taken

For regression results are:

Independent of the number of features, the break-even between parallel lleaves and 1 job seems to be around 1k samples at once, independent of the number of features. Using this logic, we would get better performance than LGBM at all number of samples.

For classification:

Also, here, the break-even is around 1k samples.

For classification with HIGHLY IMBALANCED data (1/50 positive), the break-even is only at 10k samples - Any ideas on why this is the case?

Hello, thank you for this great library. I have a question about the model cache file. I am using Ray to manage a small cluster of PCs with both Intel/AMD CPUs, and different OS (Ubuntu/ClearLinux). My program has been using numba to speed things up, and the JIT mode (instead AIT mode) works fine. Ray can send the numba functions to different PCs in the cluster and they compile locally.

So for lleaves, if I compile the models on one node, and distribute the generated cache file to all nodes in the cluster, will it work? or I have to stick to the "JIT" mode, where models are always compiled locally each time? I am using ensemble methods with many lgbm models (total >1000, each is small about 100 trees, max_depth 10). Or maybe I should have all models compiled locally on each PC? Thank you.

Hi Simon, many thanks for the nice work. I have a question about using the C_API:

If I have 2 LightGBM models in my application, and I want to predict using the C_API. I might need to have following two functions:

void forest_root_model1(double *, double *, int, int);

void forest_root_model2(double *, double *, int, int);

Do I need to modify the llvm_model.compile() function to change the function names?

Recently, I find that one of my model will cause core dump if I use lleaves for predict.

I am confused about two functions below.

In codegen.py, function param type can be int* if param is categorical

def make_tree(tree):
    # declare the function for this tree
    func_dtypes = (INT_CAT if f.is_categorical else DOUBLE for f in tree.features)
    scalar_func_t = ir.FunctionType(DOUBLE, func_dtypes)
    tree_func = ir.Function(module, scalar_func_t, name=str(tree))
    tree_func.linkage = "private"
    # populate function with IR
    gen_tree(tree, tree_func)
    return LTree(llvm_function=tree_func, class_id=tree.class_id)

But in data_processing.py with predict used, all feature param are convert to double*

def ndarray_to_ptr(data: np.ndarray):
    """
    Takes a 2D numpy array, converts to float64 if necessary and returns a pointer

    :param data: 2D numpy array. Copying is avoided if possible.
    :return: pointer to 1D array of dtype float64.
    """
    # ravel makes sure we get a contiguous array in memory and not some strided View
    data = data.astype(np.float64, copy=False, casting="same_kind").ravel()
    ptr = data.ctypes.data_as(POINTER(c_double))
    return ptr

Is this just like

int* predict(int* a, double* b);
double a = 1.1;
double b = 2.2;
predict(&a, &b);

Does this will happy in lleaves?

I find that compile can only use one cpu core. For my model, it may take long time to compile.

Can make compile with multiple threads, just like make -j ?

Model currently requires the path to a model file. I was wondering if it'd make sense to also accept a booster. We could call to_string and save it as a temporary file or just work with the string representation directly. It'd make users' life (a little) easier.

Bumps pypa/gh-action-pypi-publish from 1.5.2 to 1.6.4.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Thanks for your great work. I want to try it out, but the memory consumption is staggering. I can't use it on a machine with 32G memory, out of memory will occur Is there something I haven't set right?

import lleaves
import pdb

MODEL_TXT_PATH = '/home/kk/models/lgb080401_amp.txt'
llvm_model = lleaves.Model(model_file=MODEL_TXT_PATH)
# num = llvm_model.num_trees()
# pdb.set_trace()
llvm_model.compile(cache='./lleaves.bin', fblocksize=34)

Is there a way to effectively check if compiled models are able to run on a machine?

I am running predictions on various platforms, when loading the compiled model, I load the one which was compiled on the same platform (using: PLATFORM = sys.platform + '-' + sysconfig.get_platform().split('-')[-1].lower(), resulting in either darwin-arm64 or linux-x86_64). However sometimes models which are compiled in a linux-x86_64 environment, are not interoperable with other linux-x86_64 machines (I use AWS Fargate, which runs the container on whatever hardware is available). This results in exit code 132 (Illegal Instruction) in the model.predict() loop.

The underlying reason is probably that the underlying machines are not the same architecture (ARM based?). For example, when I compile a model within a Docker container (with DOCKER_DEFAULT_PLATFORM=linux/amd64) on my M1 Mac, it registers the platform as linux-x86_64, but the model cannot be used on AWS linux machine using Docker.

What would be a solid way to go about this issue? Is there some LLVM version which I need to look at in order for models to be interoperable?

Thanks a lot.

I am loading a model with thousands of trees, which takes approx. 10minutes. Therefore I want to compile the model once and then serialize it to file. Pickle or dill give the following error: "ValueError: ctypes objects containing pointers cannot be pickled". Is there a way to save/load the file to/from disk? Thanks :)

Forgive me, I’m ignorant of compilation nuances on different operating systems. Is windows support on PyPi possible? I see that windows support was removed in a PR about a year ago, but there are no notes.