Spline is a tool that is capable of running locally as well as part of well known pipelines like Jenkins (Jenkinsfile), Travis CI (.travis.yml) or similar ones.

Overview

Welcome to spline - the pipeline tool

PyPI version PyPI format PyPI versions PyPI license Read The Docs Coverage Status BCH compliance Known Vulnerabilities


Important note:

Since change in my job I didn't had the chance to continue on this project. My main new project is here https://github.com/thomas-lehmann-private/hyperion-task-processor which is planned to be a improved task processing tool; step by step those one will be capable to have more functionality across all platforms (powershell, batch, bash, JShell, Groovy, Kotlin, Docker, ... features depend on platform). It's fully written in Java. Stay tuned.


Table Of Content:
Motivation
Quickstart
Example
Matrix
The Model
The Pipeline
Pipeline Stages
Tasks
The Shell
Environment Variables
Docker Container
Hooks
Include
Event Logging
Tool: spline-loc
How to contact?
Links

Features:

  • Python support for 2.7.x, 3.4.x, 3.5.x, 3.6.x, PyPy and PyPy3
  • automatic schema validation for yaml file
  • matrix based pipeline with tags
  • ordered and parallel pipelines (matrix)
  • ordered and parallel tasks execution
  • pipeline stages (named groups)
  • shell script execution: inline and file
  • environment variables merged across each level: matrix, pipeline, stage, and tasks
  • support for model data (a dictionary of anything you need)
  • cleanup hook
  • filtered execution via tags
  • supporting Jinja templating in Bash scripts (also nested inside model)
  • support for Docker containers and Docker images
  • support for the Packer tool
  • execution time on each level: pipeline, stage, tasks and shell (event logging)
  • documentation here and also at read the docs
  • usable by Jenkinsfile as well as by a .travis.yml (or other pipelines).
  • dry run and debug support
  • support for Python scripts
  • support for task variables
  • support for conditional tasks
  • enabled for code reuse: !include statement

How to contact?

Interesting Links

Comments
  • Recursive render fails on raw/endraw

    Recursive render fails on raw/endraw

    Following problem:

    $ PYTHONPATH=$PWD scripts/spline --definition=examples/docker.yaml --tags=remove-docker-container
    2018-08-16 05:50:04,315 - spline.application - Running with Python 2.7.13 (default, Nov 24 2017, 17:33:09) [GCC 6.3.0 20170516]
    2018-08-16 05:50:04,321 - spline.application - Running on platform Linux-4.9.0-6-amd64-x86_64-with-debian-9.4
    2018-08-16 05:50:04,321 - spline.application - Current cpu count is 4
    2018-08-16 05:50:04,321 - spline.application - Processing pipeline definition 'examples/docker.yaml'
    2018-08-16 05:50:04,339 - spline.application - Schema validation for 'examples/docker.yaml' succeeded
    2018-08-16 05:50:04,382 - spline.tools.version - Using tool 'Bash', Version(4.4.12)
    2018-08-16 05:50:04,382 - spline.tools.version - Using tool 'Docker', Version(17.12.1)
    2018-08-16 05:50:04,383 - spline.tools.version - Using tool 'Spline', Version(1.11)
    2018-08-16 05:50:04,383 - spline.components.stage - Processing pipeline stage 'example'
    2018-08-16 05:50:04,384 - spline.components.tasks - Processing group of tasks (parallel=no)
    2018-08-16 05:50:04,386 - spline.components.tasks - Processing Bash code: start
    2018-08-16 05:50:04,397 - spline.tools.filters - render(syntax error): unexpected '.'
    2018-08-16 05:50:04,398 - spline.components.tasks - Pipeline has failed: leaving as soon as possible!
    

    The yaml contains {% raw %} and {% endraw %} which seems to be rendered twice and second time the remaining string contains {{.ID}} which cannot be resolved.

    When I remove the recursive replace all is fine.

    bug 
    opened by Nachtfeuer 7
  • Allow renderable `with` content.

    Allow renderable `with` content.

    This patch permits the with option, of tasks, to contain Jinja2 renderable content. This is useful when the models section contains data to be reused across a number of tasks.

    opened by jbenden 5
  • spline-stash: stash functionality (spline-server, part one)

    spline-stash: stash functionality (spline-server, part one)

    On the path to #63 (brainstorming: spline-server) my idea is to start as soon as possible using written functionality which is currently #64 (InMemoryFiles). With this I would like to introduce a new tool spline-stash that can store a path (recursively) at the stash-server and of course I would like to get back the content (usually on another location).

    I still have to make my thoughts about the command line syntax but I'm a bit inspired by git stash of course. The server can run either locally or at a server; the only thing you would have to do is to install spline as usual and to run the server. For the moment no configuration is involved.

    For the server I feel pretty comfortable (the moment) using Falcon (https://falconframework.org). I never used it before except running the given demo.

    enhancement 
    opened by Nachtfeuer 3
  • Conditional tasks

    Conditional tasks

    Considering that the pipeline uploads/deploys something this usually will happen on concrete branches only. As an example in some projects using Git the branch master is used as main line. Each time a pull request gets merged there a successful build should provide artifacts somewhere. The next task snippet could be the way to do it:

    - shell:
        script: echo "upload"
        when: "{{ env.BRANCH_NAME }}" == "master"
    
    enhancement 
    opened by Nachtfeuer 3
  • Asynchronous Bash execution

    Asynchronous Bash execution

    When executing a Bash script the output of it is passed back the Python tool after completion which has two effects:

    • for the time of execution of one Bash you see no output
    • the logging of the lines of the output get wrong timestamps

    It's not really a bug since the code does work.

    enhancement 
    opened by Nachtfeuer 3
  • Recursively expand Jinja2 templates

    Recursively expand Jinja2 templates

    This patch introduces automatic expansion of Jinja2 templates, except the Docker set-up related templates.

    A direct benefit of this patch is one's pipeline configuration becomes less complex; with many Jinja2 render filters used everywhere becoming unnecessary.

    opened by jbenden 2
  • Require usage of single quotes for docker -e parameter (otherwise subsitutions do happen)

    Require usage of single quotes for docker -e parameter (otherwise subsitutions do happen)

    Following problem:

    FOO="\${}" spline
    ...
    2018-06-13 15:15:08,595 - spline.components.bash - Running script /tmp/pipeline-script-Kj8mAR.sh
    2018-06-13 15:15:08,612 - spline.components.tasks -  | /tmp/pipeline-script-Kj8mAR.sh: line 24: FOO=${}: bad substitution
    2018-06-13 15:15:08,612 - spline.components.bash - Exit code has been 1
    ...
    

    Simple pipeline demo for that issue (as sufficient):

    pipeline:
      - stage(demo):
          - tasks(ordered):
            - docker(container):
                script: |
                  echo "hello"
    
    bug 
    opened by Nachtfeuer 2
  • Put curly braces and quotes on variables for Docker mounts

    Put curly braces and quotes on variables for Docker mounts

    As I were told it seems that docker run --rm -v $PWD:/mnt/host -it centos:7 bash -c "ls /mnt/host" has failed on OSX while using ${PWD} should have been working. The docker container template is required to be updated for it.

    bug 
    opened by Nachtfeuer 2
  • docker(image) task: failed to add files to image

    docker(image) task: failed to add files to image

    Because final scripts runs on temporary path Dockerfile commands like ADD are not able to fetch content from the source path.

    It turns out that providing the source path wouldn't help because Docker doesn't allow access to path outside the context (relative to the path where the build process is running)

    bug 
    opened by Nachtfeuer 2
  • New task: ansible(simple)

    New task: ansible(simple)

    • embedded solution
    • generator for one environment (inventory, group_vars, playbook)
    • dev, qa, or prod via model
    • script section is playbook
    • no roles
    • updating the presentation
    enhancement 
    opened by Nachtfeuer 2
  • Adding ssh agent support for Docker container

    Adding ssh agent support for Docker container

    • Automatically organize mount and environment variable when ssh agent is active only.
    • Basic procedure: -v $SSH_AUTH_SOCK:/ssh-agent --env SSH_AUTH_SOCK=/ssh-agent
    enhancement 
    opened by Nachtfeuer 2
  • Bump wheel from 0.29.0 to 0.38.1 in /examples/python/primes

    Bump wheel from 0.29.0 to 0.38.1 in /examples/python/primes

    Bumps wheel from 0.29.0 to 0.38.1.

    Changelog

    Sourced from wheel's changelog.

    Release Notes

    UNRELEASED

    • Updated vendored packaging to 22.0

    0.38.4 (2022-11-09)

    • Fixed PKG-INFO conversion in bdist_wheel mangling UTF-8 header values in METADATA (PR by Anderson Bravalheri)

    0.38.3 (2022-11-08)

    • Fixed install failure when used with --no-binary, reported on Ubuntu 20.04, by removing setup_requires from setup.cfg

    0.38.2 (2022-11-05)

    • Fixed regression introduced in v0.38.1 which broke parsing of wheel file names with multiple platform tags

    0.38.1 (2022-11-04)

    • Removed install dependency on setuptools
    • The future-proof fix in 0.36.0 for converting PyPy's SOABI into a abi tag was faulty. Fixed so that future changes in the SOABI will not change the tag.

    0.38.0 (2022-10-21)

    • Dropped support for Python < 3.7
    • Updated vendored packaging to 21.3
    • Replaced all uses of distutils with setuptools
    • The handling of license_files (including glob patterns and default values) is now delegated to setuptools>=57.0.0 (#466). The package dependencies were updated to reflect this change.
    • Fixed potential DoS attack via the WHEEL_INFO_RE regular expression
    • Fixed ValueError: ZIP does not support timestamps before 1980 when using SOURCE_DATE_EPOCH=0 or when on-disk timestamps are earlier than 1980-01-01. Such timestamps are now changed to the minimum value before packaging.

    0.37.1 (2021-12-22)

    • Fixed wheel pack duplicating the WHEEL contents when the build number has changed (#415)
    • Fixed parsing of file names containing commas in RECORD (PR by Hood Chatham)

    0.37.0 (2021-08-09)

    • Added official Python 3.10 support
    • Updated vendored packaging library to v20.9

    ... (truncated)

    Commits
    • 6f1608d Created a new release
    • cf8f5ef Moved news item from PR #484 to its proper place
    • 9ec2016 Removed install dependency on setuptools (#483)
    • 747e1f6 Fixed PyPy SOABI parsing (#484)
    • 7627548 [pre-commit.ci] pre-commit autoupdate (#480)
    • 7b9e8e1 Test on Python 3.11 final
    • a04dfef Updated the pypi-publish action
    • 94bb62c Fixed docs not building due to code style changes
    • d635664 Updated the codecov action to the latest version
    • fcb94cd Updated version to match the release
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
  • Bump wheel from 0.29.0 to 0.38.1

    Bump wheel from 0.29.0 to 0.38.1

    Bumps wheel from 0.29.0 to 0.38.1.

    Changelog

    Sourced from wheel's changelog.

    Release Notes

    UNRELEASED

    • Updated vendored packaging to 22.0

    0.38.4 (2022-11-09)

    • Fixed PKG-INFO conversion in bdist_wheel mangling UTF-8 header values in METADATA (PR by Anderson Bravalheri)

    0.38.3 (2022-11-08)

    • Fixed install failure when used with --no-binary, reported on Ubuntu 20.04, by removing setup_requires from setup.cfg

    0.38.2 (2022-11-05)

    • Fixed regression introduced in v0.38.1 which broke parsing of wheel file names with multiple platform tags

    0.38.1 (2022-11-04)

    • Removed install dependency on setuptools
    • The future-proof fix in 0.36.0 for converting PyPy's SOABI into a abi tag was faulty. Fixed so that future changes in the SOABI will not change the tag.

    0.38.0 (2022-10-21)

    • Dropped support for Python < 3.7
    • Updated vendored packaging to 21.3
    • Replaced all uses of distutils with setuptools
    • The handling of license_files (including glob patterns and default values) is now delegated to setuptools>=57.0.0 (#466). The package dependencies were updated to reflect this change.
    • Fixed potential DoS attack via the WHEEL_INFO_RE regular expression
    • Fixed ValueError: ZIP does not support timestamps before 1980 when using SOURCE_DATE_EPOCH=0 or when on-disk timestamps are earlier than 1980-01-01. Such timestamps are now changed to the minimum value before packaging.

    0.37.1 (2021-12-22)

    • Fixed wheel pack duplicating the WHEEL contents when the build number has changed (#415)
    • Fixed parsing of file names containing commas in RECORD (PR by Hood Chatham)

    0.37.0 (2021-08-09)

    • Added official Python 3.10 support
    • Updated vendored packaging library to v20.9

    ... (truncated)

    Commits
    • 6f1608d Created a new release
    • cf8f5ef Moved news item from PR #484 to its proper place
    • 9ec2016 Removed install dependency on setuptools (#483)
    • 747e1f6 Fixed PyPy SOABI parsing (#484)
    • 7627548 [pre-commit.ci] pre-commit autoupdate (#480)
    • 7b9e8e1 Test on Python 3.11 final
    • a04dfef Updated the pypi-publish action
    • 94bb62c Fixed docs not building due to code style changes
    • d635664 Updated the codecov action to the latest version
    • fcb94cd Updated version to match the release
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
  • [Snyk] Security upgrade setuptools from 39.0.1 to 65.5.1

    [Snyk] Security upgrade setuptools from 39.0.1 to 65.5.1

    This PR was automatically created by Snyk using the credentials of a real user.


    Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.

    Changes included in this PR

    • Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
      • requirements.txt
    ⚠️ Warning
    sphinxcontrib-inheritance 0.9.0 has requirement Sphinx<=1.4.9, but you have Sphinx 1.6.5.
    Sphinx 1.6.5 requires Pygments, which is not installed.
    astroid 1.6.6 requires lazy-object-proxy, which is not installed.
    
    

    Vulnerabilities that will be fixed

    By pinning:

    Severity | Priority Score (*) | Issue | Upgrade | Breaking Change | Exploit Maturity :-------------------------:|-------------------------|:-------------------------|:-------------------------|:-------------------------|:------------------------- low severity | 441/1000
    Why? Recently disclosed, Has a fix available, CVSS 3.1 | Regular Expression Denial of Service (ReDoS)
    SNYK-PYTHON-SETUPTOOLS-3113904 | setuptools:
    39.0.1 -> 65.5.1
    | No | No Known Exploit

    (*) Note that the real score may have changed since the PR was raised.

    Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.

    Check the changes in this PR to ensure they won't cause issues with your project.


    Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.

    For more information: 🧐 View latest project report

    πŸ›  Adjust project settings

    πŸ“š Read more about Snyk's upgrade and patch logic


    Learn how to fix vulnerabilities with free interactive lessons:

    πŸ¦‰ Regular Expression Denial of Service (ReDoS)

    opened by Nachtfeuer 0
  • [Snyk] Security upgrade Pillow from 6.2.2 to 8.2.0

    [Snyk] Security upgrade Pillow from 6.2.2 to 8.2.0

    Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.

    Changes included in this PR

    • Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
      • requirements.txt
    ⚠️ Warning
    sphinxcontrib-inheritance 0.9.0 has requirement Sphinx<=1.4.9, but you have Sphinx 1.6.5.
    
    

    Vulnerabilities that will be fixed

    By pinning:

    Severity | Priority Score (*) | Issue | Upgrade | Breaking Change | Exploit Maturity :-------------------------:|-------------------------|:-------------------------|:-------------------------|:-------------------------|:------------------------- high severity | 661/1000
    Why? Recently disclosed, Has a fix available, CVSS 7.5 | Out-of-bounds Read
    SNYK-PYTHON-PILLOW-1292150 | Pillow:
    6.2.2 -> 8.2.0
    | No | No Known Exploit high severity | 661/1000
    Why? Recently disclosed, Has a fix available, CVSS 7.5 | Out-of-bounds Read
    SNYK-PYTHON-PILLOW-1292151 | Pillow:
    6.2.2 -> 8.2.0
    | No | No Known Exploit

    (*) Note that the real score may have changed since the PR was raised.

    Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the effected dependencies could be upgraded.

    Check the changes in this PR to ensure they won't cause issues with your project.


    Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.

    For more information: 🧐 View latest project report

    πŸ›  Adjust project settings

    πŸ“š Read more about Snyk's upgrade and patch logic

    opened by snyk-bot 0
  • [Snyk] Security upgrade Pillow from 6.2.2 to 8.2.0

    [Snyk] Security upgrade Pillow from 6.2.2 to 8.2.0

    Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.

    Changes included in this PR

    • Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
      • requirements.txt
    ⚠️ Warning
    sphinxcontrib-inheritance 0.9.0 has requirement Sphinx<=1.4.9, but you have Sphinx 1.6.5.
    
    

    Vulnerabilities that will be fixed

    By pinning:

    Severity | Priority Score (*) | Issue | Upgrade | Breaking Change | Exploit Maturity :-------------------------:|-------------------------|:-------------------------|:-------------------------|:-------------------------|:------------------------- high severity | 661/1000
    Why? Recently disclosed, Has a fix available, CVSS 7.5 | Denial of Service (DoS)
    SNYK-PYTHON-PILLOW-1090584 | Pillow:
    6.2.2 -> 8.2.0
    | No | No Known Exploit high severity | 661/1000
    Why? Recently disclosed, Has a fix available, CVSS 7.5 | Denial of Service (DoS)
    SNYK-PYTHON-PILLOW-1090586 | Pillow:
    6.2.2 -> 8.2.0
    | No | No Known Exploit high severity | 661/1000
    Why? Recently disclosed, Has a fix available, CVSS 7.5 | Denial of Service (DoS)
    SNYK-PYTHON-PILLOW-1090587 | Pillow:
    6.2.2 -> 8.2.0
    | No | No Known Exploit high severity | 661/1000
    Why? Recently disclosed, Has a fix available, CVSS 7.5 | Denial of Service (DoS)
    SNYK-PYTHON-PILLOW-1090588 | Pillow:
    6.2.2 -> 8.2.0
    | No | No Known Exploit

    (*) Note that the real score may have changed since the PR was raised.

    Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the effected dependencies could be upgraded.

    Check the changes in this PR to ensure they won't cause issues with your project.


    Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.

    For more information: 🧐 View latest project report

    πŸ›  Adjust project settings

    πŸ“š Read more about Snyk's upgrade and patch logic

    opened by snyk-bot 0
  • [Snyk] Security upgrade Pygments from 2.5.2 to 2.7.4

    [Snyk] Security upgrade Pygments from 2.5.2 to 2.7.4

    Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.

    Changes included in this PR

    • Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
      • requirements.txt
    ⚠️ Warning
    sphinxcontrib-inheritance 0.9.0 has requirement Sphinx<=1.4.9, but you have Sphinx 1.6.5.
    Sphinx 1.6.5 requires Pygments, which is not installed.
    
    

    Vulnerabilities that will be fixed

    By pinning:

    Severity | Priority Score (*) | Issue | Upgrade | Breaking Change | Exploit Maturity :-------------------------:|-------------------------|:-------------------------|:-------------------------|:-------------------------|:------------------------- high severity | 661/1000
    Why? Recently disclosed, Has a fix available, CVSS 7.5 | Denial of Service (DoS)
    SNYK-PYTHON-PYGMENTS-1088505 | Pygments:
    2.5.2 -> 2.7.4
    | No | No Known Exploit

    (*) Note that the real score may have changed since the PR was raised.

    Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the effected dependencies could be upgraded.

    Check the changes in this PR to ensure they won't cause issues with your project.


    Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.

    For more information: 🧐 View latest project report

    πŸ›  Adjust project settings

    πŸ“š Read more about Snyk's upgrade and patch logic

    opened by snyk-bot 0
Releases(1.12)
  • 1.12(Oct 26, 2018)

    You can install it with following command (for upgrade please add --upgrade):

    pip install spline
    

    One feature only: labels for docker(container) task.

    Source code(tar.gz)
    Source code(zip)
  • 1.11(Sep 3, 2018)

    You can install it with following command (for upgrade please add --upgrade):

    pip install spline
    
    • spline-loc tool (https://spline.readthedocs.io/en/latest/spline_loc.html)
    • bugfix for tool versions using spline on OSX
    • bugfix for escaping values of Bash variables
    • bugfix for closing of file handles (thanks to @jbenden)
    • bugfix for exec permission for user only (thanks to @jbenden)
    • extension to with field being renderable (thanks to @jbenden)
    Source code(tar.gz)
    Source code(zip)
  • 1.10.2(May 7, 2018)

    You can install it with following command (for upgrade please add --upgrade):

    pip install spline
    
    • break pipeline when required tools are missing
    • code reuse: !include support for main yaml
    • exclude copying some path variables (HOME, PYTHONPATH, JAVAHOME, PATH, ...)
    • rendering support for task title
    • some minor bugfixes (including quotes and curly braces for $PWD)
    • documentation and presentation extended
    Source code(tar.gz)
    Source code(zip)
  • 1.9(Apr 8, 2018)

    You can install it with following command (for upgrade please add --upgrade):

    pip install spline
    
    • new task: ansible(simple)
    • new task: packer
    • bugfix: variables field has not been available in templating (Jinja2) for Docker images
    • bugfix: path adjustment for Dockerfile to be in $PWD so that you can use ADD in Dockerfile for current source path
    • improvement: for ordered task execution enable immediate variable evaluation on next task
    • improvement: new optional field network for Docker container task
    • documentation as presentation updated (as usual)
    Source code(tar.gz)
    Source code(zip)
  • 1.8(Mar 11, 2018)

    You can install it with following command (for upgrade please add --upgrade):

    pip install spline
    
    • Bash strict mode (--strict)
    • Temporary scripts path adjustable (--temporary-scripts-path)
    • Forwarding SSH agent into Docker container
    • Fixed exit code (variable) for Python tasks
    • Documentation extended for Python development: http://spline.readthedocs.io/en/latest/development.html
    • Internal: Testing wheel file installation after build
    Source code(tar.gz)
    Source code(zip)
  • 1.7.2(Feb 1, 2018)

    You can install it with following command (for upgrade please add --upgrade):

    pip install spline
    

    Following content:

    • HTML report generation (with auto refresh) Details: http://spline.readthedocs.io/en/latest/report.html
    • tasks variables (writing output of a task into a variable being known py whole pipeline) Details: http://spline.readthedocs.io/en/latest/tasks.html#variables-on-tasks
    • condition tasks (allowing jinja templating and some concrete simple conditions) Details: http://spline.readthedocs.io/en/latest/conditions.html
    • You now can leave out --definition when you use a pipeline.yaml (it's now default)
    • Contribution guide added
    Source code(tar.gz)
    Source code(zip)
  • 1.6.1(Jan 6, 2018)

    You can install/upgrade it with following command:

    pip install spline [--upgrade]
    

    Following content:

    • New option --dry-run which does show all Bash script content instead of executing it
    • New option --debug which adds set -x to Bash script (verbose logging of Bash commands)
    • The tools now does log used cpu count
    • To some extend improved handling for unicode
    • Handling of Python 3.7 added to project (pipeline.yaml and examples/docker-image.yaml). Adding Python 3.7 to .travis.yml probably won't work since I detected issues with pylint and again with unicode. I tested following exact version: https://www.python.org/downloads/release/python-370a2/. Since it is called a "early developer preview" nothing to worry about yet.
    Source code(tar.gz)
    Source code(zip)
  • 1.5(Dec 27, 2017)

    You can install/upgrade it with following command:

    pip install spline [--upgrade]
    

    Following content:

    • New feature: Python task; read here: http://spline.readthedocs.io/en/latest/python.html
    • Better Code 10/10
    • Support for Pypy (2.7.x compatible) and Pypy3 (3.5.x compatible)
    • Bandit analyzer added
    • README (with badges) now also on PyPI
    • Code Of Conduct added
    Source code(tar.gz)
    Source code(zip)
  • 1.4(Dec 22, 2017)

    You can install/upgrade it with following command:

    pip install spline [--upgrade]
    

    Following content:

    • code complexity of spline reduced (radon and flake8)
    • coverage now clearly above 90% (roughly 94%)
    • pipeline definition for spline itself added (docker tests will be skipped there)
    • new feature docker(image)
    Source code(tar.gz)
    Source code(zip)
  • 1.3(Dec 20, 2017)

    You can install it with following command:

    pip install spline --upgrade
    
    • added with field for shell and docker container allowing to run same task with different data
    • fixed asynchronous bash execution
    • improved schema by replacing quite nice pykwalify with a bit more Pythonic schema solution being more independent from file format.
    Source code(tar.gz)
    Source code(zip)
  • 1.2(Dec 10, 2017)

    You can install it with following command:

    pip install spline --upgrade
    

    One main task was to have unitests in place. The project is currently at 86% code coverage. Another main task has been to provide a realistic demo on how to build a project with spline + Python + tox.

    Please check https://github.com/Nachtfeuer/pipeline/milestone/3?closed=1 for details. In addition one problem with PIPELINE_BASH_FILE and availability of model and env inside Docker container.

    The documentation of the first example can be seen here:

    • https://github.com/Nachtfeuer/pipeline/blob/master/docs/example.rst
    • or here: http://spline.readthedocs.io/en/latest/example.html
    Source code(tar.gz)
    Source code(zip)
Owner
Thomas Lehmann
Highly interested in development, programming languages, infrastructure, processes and design. Favor C++, Java and Python.
Thomas Lehmann
Regulatory Instruments for Fair Personalized Pricing.

Fair pricing Source code for WWW 2022 paper Regulatory Instruments for Fair Personalized Pricing. Installation Requirements Linux with Python = 3.6 p

Renzhe Xu 6 Oct 26, 2022
Scikit-event-correlation - Event Correlation and Forecasting over High Dimensional Streaming Sensor Data algorithms

scikit-event-correlation Event Correlation and Changing Detection Algorithm Theo

Intellia ICT 5 Oct 30, 2022
A Haskell kernel for IPython.

IHaskell You can now try IHaskell directly in your browser at CoCalc or mybinder.org. Alternatively, watch a talk and demo showing off IHaskell featur

Andrew Gibiansky 2.4k Dec 29, 2022
Implicit Model Specialization through DAG-based Decentralized Federated Learning

Federated Learning DAG Experiments This repository contains software artifacts to reproduce the experiments presented in the Middleware '21 paper "Imp

Operating Systems and Middleware Group 5 Oct 16, 2022
The World of an Octopus: How Reporting Bias Influences a Language Model's Perception of Color

The World of an Octopus: How Reporting Bias Influences a Language Model's Perception of Color Overview Code and dataset for The World of an Octopus: H

1 Nov 13, 2021
Make differentially private training of transformers easy for everyone

private-transformers This codebase facilitates fast experimentation of differentially private training of Hugging Face transformers. What is this? Why

Xuechen Li 73 Dec 28, 2022
Dense Prediction Transformers

Vision Transformers for Dense Prediction This repository contains code and models for our paper: Vision Transformers for Dense Prediction RenΓ© Ranftl,

Intelligent Systems Lab Org 1.3k Jan 02, 2023
In this project we predict the forest cover type using the cartographic variables in the training/test datasets.

Kaggle Competition: Forest Cover Type Prediction In this project we predict the forest cover type (the predominant kind of tree cover) using the carto

Marianne Joy Leano 1 Mar 15, 2022
Code repository for the paper Computer Vision User Entity Behavior Analytics

Computer Vision User Entity Behavior Analytics Code repository for "Computer Vision User Entity Behavior Analytics" Code Description dataset.csv As di

Sameer Khanna 2 Aug 20, 2022
A curated list of awesome resources related to Semantic SearchπŸ”Ž and Semantic Similarity tasks.

A curated list of awesome resources related to Semantic SearchπŸ”Ž and Semantic Similarity tasks.

224 Jan 04, 2023
Implementations of CNNs, RNNs, GANs, etc

Tensorflow Programs and Tutorials This repository will contain Tensorflow tutorials on a lot of the most popular deep learning concepts. It'll also co

Adit Deshpande 1k Dec 30, 2022
Visualizing lattice vibration information from phonon dispersion to atoms (For GPUMD)

Phonon-Vibration-Viewer (For GPUMD) Visualizing lattice vibration information from phonon dispersion for primitive atoms. In this tutorial, we will in

Liangting 6 Dec 10, 2022
Pytorch implementation of the paper SPICE: Semantic Pseudo-labeling for Image Clustering

SPICE: Semantic Pseudo-labeling for Image Clustering By Chuang Niu and Ge Wang This is a Pytorch implementation of the paper. (In updating) SOTA on 5

Chuang Niu 154 Dec 15, 2022
Gems & Holiday Package Prediction

Predictive_Modelling Gems & Holiday Package Prediction This project is based on 2 cases studies : Gems Price Prediction and Holiday Package prediction

Avnika Mehta 1 Jan 27, 2022
Predicting future trajectories of people in cameras of novel scenarios and views.

Pedestrian Trajectory Prediction Predicting future trajectories of pedestrians in cameras of novel scenarios and views. This repository contains the c

8 Sep 03, 2022
Automatic voice-synthetised summaries of latest research papers on arXiv

PaperWhisperer PaperWhisperer is a Python application that keeps you up-to-date with research papers. How? It retrieves the latest articles from arXiv

Valerio Velardo 124 Dec 20, 2022
MapReader: A computer vision pipeline for the semantic exploration of maps at scale

MapReader A computer vision pipeline for the semantic exploration of maps at scale MapReader is an end-to-end computer vision (CV) pipeline designed b

Living with Machines 25 Dec 26, 2022
190 Jan 03, 2023
Learning to Draw: Emergent Communication through Sketching

Learning to Draw: Emergent Communication through Sketching This is the official code for the paper "Learning to Draw: Emergent Communication through S

19 Jul 22, 2022
EdiBERT, a generative model for image editing

EdiBERT, a generative model for image editing EdiBERT is a generative model based on a bi-directional transformer, suited for image manipulation. The

16 Dec 07, 2022