TODOs:

• [x] Update docs with new arguments & explain vectorized feature function behavior
• [x] Benchmark runtime of vectorized functions
• [x] Benchmark memory peak of vectorized functions
• [x] Add proper checks, with appropriate error msg, for equally sampled data assumption
• [x] Add tests

:recycle: Refactor indexing

• :bug: fix bug with vectorized=True for strided rolling
  • [x] vectorized support for single feature windows
  • [x] vectorized support for empty feature windows
  • [x] test the above
• :recycle: refactor the strided window segmentation (& indexing)
  • [x] add include_final_window argument to FeatureCollection .calculate (& StridedRolling)
  • [x] update docs
  • [x] test the above
  • [ ] remove support for TimeSequenceStridedRolling => Decided to not do this in this PR. We will leave this for another PR.
• :see_no_evil: undo breaking change from #62 (to make code backwards compatible)
  • [x] revert the default window_idx argument in FeatureCollection.calculate to "end"
• :robot: extend test matrix & update dependencies
  • [x] Add Python 3.10 to the test matrix
  • [x] Update dependencies (& remove locked statsmodels dependency - https://github.com/blue-yonder/tsfresh/issues/897)

:scissors: Decouple stride

We (@jonasvdd, @emield12, and @jvdd) believe that the stride should not be hardly coupled with a FeatureDescriptor. Therefore, to make tsflex more flexible (:wink:) we make the stride argument optional for FeatureDescriptor and MultipleFeatureDescriptor and add the functionality to pass your stride(s) to the FeatureCollection.calculate method.

• :eyes: externally visible changes
  • [x] make stride optional in FeatureDescriptor
  • [x] add stride argument to FeatureCollection.calculate
  • [x] FeatureDescriptor / FeatureCollection.calculate should accept multiple strides
    • [x] StridedRolling should accept multiple strides
    • [x] test the above
  • [x] remove stride from output column name
    • [x] update reduce method
    • [x] update reduce method tests
  • [x] update the logging to handle multiple strides
    • [x] extend logging tests
• :package: internal changes
  • [x] change stride -> strides: which is either a list of stride sizes (float or pd.Timedelta) or None (in StridedRolling and StridedRollingFactory)
  • [x] identify feature descriptors (FD) based on their window - output names
    • [x] set interection after StridedRolling search sorted TODO: moet geoptimaliseerd worden

:sparkles: Support setpoints

• [x] support setpoints
• [x] test the above => Note: we allow setpoints of different timezones as the np.datetime64 conversion of these allow comparison..
• [ ] trim range if not in data => Decided to not do this! (as this is somewhat an ambiguous operation) As using segment indexes is already an advanced operation, it is the user its responsability to either trim the segmented indexes or make their features robust.

:see_no_evil: other stuff

• :bug: fix bug with features.logger that does not handle numeric window & strides
  • [x] improve parsing for window and stride values in _parse_logging_execution_to_df
  • [x] test the above
• :eyes: other minor stuff
  • [x] support offline data load
  • [x] warn the user with a RuntimeWarining when the data its index (passed to FeatureCollection.calculate) is not monotonically increasing
  • [x] test the above

The closed argument was set to None which is the default, removing it should thus not have any impact. Since Pandas 1.4.0 this argument has been deprecated in favour of the inclusive argument so you get a lot of warnings when running the code. The default argument to inclusive is "both" which has the same behaviour as the current code. I thus see no need to add it. https://pandas.pydata.org/docs/reference/api/pandas.date_range.html

This PR handles

(1) :bug: a bug in the bound_method + sequence based strided rolling

• [x] check if .loc induces memory peak
• [x] agree on what behavior is preferred for segmentation indexing
• Agreed behavior:
  • make window_idx="begin" default instead of "end"
  • sequences should be segmented into n segments if there are exactly n segments possible (e.g., window=2, stride=2 => 5 segments on sequence of length 10)
  • we remain the current behavior of the end-index of the segments.
• [x] extend tests

(2) :sparkles: extends integration with other feature extraction packages

• [x] add and test catch22 integration wrapper
• [x] :x: add and test scikit-learn transformer wrapper
• Decided to not do this (for now): have atm an ambiguas implementation -> more details in #56

(3): :zap: faster irregular data check

(4): :fire: add kaggle TPSAPR2022 notebook to ML examples

Hello,

I would like to generate features for each observation of my time serie and not only window by window.

Does this possibility exist in tsfex and do you know how to do it ?

Thanks in advance

This PR

• Adds Windows & MacOS testing to the matrix
• Fix a bug on Windows (PermissionError: [WinError 32]), which occurs when a logging file handler is not properly unregistered and closed
• Disable multiprocessing on Windows, see #51

Changes;

• Overall dependencies are updated
• Statsmodels dependencies are fixed @ 0.12.2 to avoid tsfresh errors. See https://github.com/blue-yonder/tsfresh/issues/897

Objectives:

Functionality

• [ ] convenient way to extract features over the whole, unsegmented data (see also #67)
  • [ ] Discuss + decide together with @jvdd @mbignotti what option seems best to serve this functionality (regarding end user perspective)

Available options:

1. introduce a new method to the FeatureCollection (as done here):
   • advantages ✅
     • Explicit method definition, less confusion for end-users
   • disadvanges :x:
     • A new method is introduced / less uniform interface to perform computation
2. Perform unsegmented feature computation when all window and/or stride are NOT set.
   • advantages ✅
     • more homogenous interface
   • disadvanges :x:
     • somewhat more implicitness code example:

 # NOTE: window and stride parameters are omitted. 
fc = FeatureCollection(
    FeatureDescriptor(
        function = np.mean,
        series_name="Value",
    )
)

# Uses the whole (unsegmented) series of `data` to 
# calculate the features. method remains the same.
fc.calculate(data=df, return_df=True)

3. Perform unsegmented feature computation when all windows are set to -1
4. a combination of (2.) and (3.)

As for now, this is performed by introducing the calculate_unsegmented method to the FeatureCollection:

Bug fixes

• [ ] fix the window_idx="end" and (window-size > data-range) bug

First of all, this package is awesome. The community that deals with time series data needed to improve the game and tsflex have everything to be the main library.

However, here are a few specific suggestions:

Remove "windows" and "strides" arguments altogether for feature extraction: It does seem a bit excessive but hear me out. They are good arguments but not fundamental for feature extraction. They could be used in data preparation, Alteryx has a library called "compose" (https://github.com/alteryx/compose) just for the purpose of creating multiple time frame windows. Once the "window" is ready, just select the functions. I propose tsflex main function (feature_collection.calculate) just use time series data and a list of functions for feature extraction, no window or strides.

Explaining further: The way I view the implementation of the essentials would be only this: feature_collection.calculate(time_series_df, functions). If any of the columns of the time series had any data type other than int, float, it could simply raise an error or ignore the column.

Window and stride also make feature_collection.reduce function hard to use: After feature selection and having selected a few columns of the many created using tsflex I use the reduce that gives me the functions for transformation/extraction. The problem is that the naming convention includes window and strides (e.g: Open__mean__w=233500_s=233500) which means I have to have a time series with the same characteristics/size, which often doesn't happen. I use the arguments windows and strides like the following:

simple_feats = MultipleFeatureDescriptors( functions=tsfresh_settings_wrapper(settings), series_names="Open", windows=len(stock_data) - 1, strides=len(stock_data) - 1, ) feature_collection = FeatureCollection(simple_feats) features_df = feature_collection.calculate( stock_full, return_df=True, show_progress=True, approve_sparsity=(True) )

I use this because I need to process the whole dataset.

Anyway, I hope this is helpful.

Hello, First of all, I would like to thank you for the really nice library. I think it is much more straight forward and at the same time flexible, compared to similar libraries. I have a use case where sometimes I need to compute features in a rolling fashion, for which the window parameter of the FeatureDescriptor object is really helpful, and some other times I need to compute features on time series batches. That is, the window parameter equals the length of the entire time series. However, I'm having a few issues with the latter case. Here is an example:

import numpy as np
import pandas as pd
from tsflex.features import FeatureDescriptor, FeatureCollection

series = np.random.rand(100)
ts_index = pd.date_range(start="2022-06-09 00:00:00", periods=len(series), freq="min")
df = pd.DataFrame({"Value": series}, index=ts_index)

fc = FeatureCollection(
    FeatureDescriptor(
        function = np.mean,
        series_name="Value",
        window=len(df),
        stride=1
    )
)

fc.calculate(data=df, return_df=True)

If I run the code above, I get the following error (personal info are hidden):

Traceback (most recent call last):
  File "/****/*****/*****/***/***/***/python3.8/site-packages/tsflex/features/feature_collection.py", line 394, in calculate
    calculated_feature_list = [self._executor(idx) for idx in idxs]
  File "/****/*****/*****/***/***/***/python3.8/site-packages/tsflex/features/feature_collection.py", line 394, in <listcomp>
    calculated_feature_list = [self._executor(idx) for idx in idxs]
  File "/****/*****/*****/***/***/***/python3.8/site-packages/tsflex/features/feature_collection.py", line 208, in _executor
    stroll, function = get_stroll_func(idx)
  File "/****/*****/*****/***/***/***/python3.8/site-packages/tsflex/features/feature_collection.py", line 245, in get_stroll_function
    stroll = StridedRollingFactory.get_segmenter(**stroll_arg_dict)
  File "/****/*****/*****/***/***/***/python3.8/site-packages/tsflex/features/segmenter/strided_rolling_factory.py", line 75, in get_segmenter
    return TimeIndexSampleStridedRolling(data, window, stride, **kwargs)
  File "/****/*****/*****/***/***/***/python3.8/site-packages/tsflex/features/segmenter/strided_rolling.py", line 495, in __init__
    super().__init__(series_list, window, stride, *args, **kwargs)
  File "/****/*****/*****/***/***/***/python3.8/site-packages/tsflex/features/segmenter/strided_rolling.py", line 373, in __init__
    super().__init__(data, window, stride, *args, **kwargs)
  File "/****/*****/*****/***/***/***/python3.8/site-packages/tsflex/features/segmenter/strided_rolling.py", line 147, in __init__
    if np.ptp(container.end_indexes - container.start_indexes) != 0:
  File "<__array_function__ internals>", line 180, in ptp
  File "/****/*****/*****/***/***/***/python3.8/site-packages/numpy/core/fromnumeric.py", line 2667, in ptp
    return _methods._ptp(a, axis=axis, out=out, **kwargs)
  File "/****/*****/*****/***/***/***/python3.8/site-packages/numpy/core/_methods.py", line 278, in _ptp
    umr_maximum(a, axis, None, out, keepdims),
ValueError: zero-size array to reduction operation maximum which has no identity
---------------------------------------------------------------------------
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/***/*****/****/****/***/***/python3.8/site-packages/tsflex/features/feature_collection.py", line 418, in calculate
    raise RuntimeError(
RuntimeError: Feature Extraction halted due to error while extracting one (or multiple) feature(s)! See stack trace above.

If I specify window=len(df) - 1,it works but then, of course, it is not using the last data point in the calculation.

Am I doing something wrong? Is there a way to achieve the required behaviour?

Thanks a lot!

Environment: python==3.8.13 numpy==1.22.4 pandas==1.4.2 tsflex==0.2.3.7.7

Current version:

new features:

• add the columns: duration % -> can be directly calculated from the duration column, so does not need to be stored within the logs itself output_names -> the output names of the adjusted / newly created series. Can help to improve the function recall?

• https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.resample.html.
• https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.rolling.html?highlight=rolling#pandas.DataFrame.rolling Maybe we can use
  • the same naming convention
  • or even re-use their underlying logic (i.e., don't reimplement the wheel)