PipeLayer is a lightweight Python pipeline framework

Overview

PipeLayer

PipeLayer is a lightweight Python pipeline framework. Define a series of steps, and chain them together to create modular applications.

Table of Contents

Installation

From the command line:

pip install pipelayer

Getting Started

Step 1: Create Pipeline Filters

hello_world_filters.py

from pipelayer import Filter


class HelloFilter(Filter):
    def run(self, data, context):
        return "Hello"


class WorldFilter(Filter):
    def run(self, data, context):
        return f"{data},  World!"

functions.py

def create_message_dict(data, context):
    return {"message": data}

Step 2: Create a Pipeline

Create a module to run the pipeline:

app.py

from pipelayer import Pipeline

from functions import create_message
from hello_world_filters import HelloFilter, WorldFilter


if __name__ = "__main__":
    hello_world_pipeline = Pipeline([
        HelloFilter,                           # pipeline.Filter type
        WorldFilter,                           # pipeline.Filter instance
        create_message_dict                    # function type
        lambda data, context: json.dumps(data) # anonymous function
    ])

    output = hello_world_pipeline.run()

    # output = '{"message": "Hello, World!"}'

    print(f"Pipeline Output: {output}")
    print(hello_world_pipeline.manifest.__dict__)

Step 3: Run the Pipeline

from the command line:

run app.py

The Framework

pipelayer.Pipeline

__init__(steps, name)

args:

  • steps: List[Union[Step, Callable[[Any, Context], Any]]]
    A list of:

    • Classes and Instances that derive from pipelayer.Filter and implement the run method

    • Classes that implement the pipelayer.Step protocol

    • Functions (instance/class/static/module) that have the following signature

      def func(data: Any, context: Any)
    • Anonymous functions (lambda) with two arguments that follow this pattern:

      my_func = lambda data, context: data
    • Instances of pipelayer.Pipeline

  • name: Optional[str]
    If not specified, the class name will be used.

Properties:

name: str

state: Pipeline.State

steps: List[Union[Step, Callable[[Any, Context], Any]]]

manifest: Manifest
An instance of pipelayer.Manifest that is created when the run method is called.

Methods:

run(data, context) -> Any
The pipeline runner that iterates through the steps and pipes filter output to the next step.

args:

  • data: Any
  • context: pipelayer.Context

pipelayer.Switch

__init__(expression, cases, name)
An implementation of a Switch statement as a pipeline filter

args:

  • expression: Union[Step, Callable[[Any, Context], Any]]
  • cases: Dict[Union[Step, Callable[[Any, Context], Any]]]
  • name: Optional[str]
    If not specified, the class name will be used.

Properties:

expression: Union[Step, Callable[[Any, Context], Any]]
cases: Dict[Union[Step, Callable[[Any, Context], Any]]]
name: Optional[str]
manifest: Manifest

Methods:

run(data, context) -> Any
The switch runner that evaluates the specified expresssion executes the matching case.

pipelayer.Filter

__init__(name, pre_process, post_process)

args:

  • name: Optional[str]
    If not specified, the class name will be used.
  • pre_process: Optional[Callable[[Any, Context], Any]
  • post_process: Optional[Callable[[Any, Context], Any]

Properties:

pre_process: Optional[Callable[[Any, Context], Any]
post_process: Optional[Callable[[Any, Context], Any]

Events:
Events are lists of callables assignable after instantiation and are raised if the pipelayer.filter.raise_events decorator is applied to the implementation of the run method.

start: List[Callable[[Filter, Any], Any]]
Raised before the run method is invoked.

exit: List[Callable[[Filter, Any], Any]]
Raised if action is set to Action.SKIP or Action.EXIT in either a start or stop event handler.

end: List[Callable[[Filter, Any], Any]]
Raised after the run method is invoked.

pipelayer.FilterEventArgs

__init__(data, context, state)

args:

  • data: Any
  • context: Context
  • state: State

pipelayer.Context

A abstract base class for runtime app data.

pipelayer.Manifest

The Manifest keeps a record of Pipeline and Filter activity.

Utilities

pipelayer.util.render_manifest(manifest, indent) -> str
Static function that renders formatted JSON data

args:

  • manifest: Manifest
  • indent: Optional[int]
    Default value is 2.
You might also like...
The Python micro framework for building web applications.

Flask Flask is a lightweight WSGI web application framework. It is designed to make getting started quick and easy, with the ability to scale up to co

Tornado is a Python web framework and asynchronous networking library, originally developed at FriendFeed.

Tornado Web Server Tornado is a Python web framework and asynchronous networking library, originally developed at FriendFeed. By using non-blocking ne

Async Python 3.6+ web server/framework | Build fast. Run fast.
Async Python 3.6+ web server/framework | Build fast. Run fast.

Sanic | Build fast. Run fast. Build Docs Package Support Stats Sanic is a Python 3.6+ web server and web framework that's written to go fast. It allow

bottle.py is a fast and simple micro-framework for python web-applications.

Bottle: Python Web Framework Bottle is a fast, simple and lightweight WSGI micro web-framework for Python. It is distributed as a single file module a

Pyramid - A Python web framework

Pyramid Pyramid is a small, fast, down-to-earth, open source Python web framework. It makes real-world web application development and deployment more

The no-nonsense, minimalist REST and app backend framework for Python developers, with a focus on reliability, correctness, and performance at scale.
The no-nonsense, minimalist REST and app backend framework for Python developers, with a focus on reliability, correctness, and performance at scale.

The Falcon Web Framework Falcon is a reliable, high-performance Python web framework for building large-scale app backends and microservices. It encou

web.py is a web framework for python that is as simple as it is powerful.

web.py is a web framework for Python that is as simple as it is powerful. Visit http://webpy.org/ for more information. The latest stable release 0.62

A familiar HTTP Service Framework for Python.
A familiar HTTP Service Framework for Python.

Responder: a familiar HTTP Service Framework for Python Powered by Starlette. That async declaration is optional. View documentation. This gets you a

The Modern And Developer Centric Python Web Framework. Be sure to read the documentation and join the Slack channel questions: http://slack.masoniteproject.com
The Modern And Developer Centric Python Web Framework. Be sure to read the documentation and join the Slack channel questions: http://slack.masoniteproject.com

NOTE: Masonite 2.3 is no longer compatible with the masonite-cli tool. Please uninstall that by running pip uninstall masonite-cli. If you do not unin

Comments
  • Python 3.7 - unable to run sample code

    Python 3.7 - unable to run sample code

    Hi,

    With Python 3.7 I cannot run sample code. I have below error:

    Traceback (most recent call last): File "C:/Github/Python/pipelayers/app.py", line 1, in from pipelayer import Pipeline File "C:\Github\Python\venv\lib\site-packages\pipelayer_init_.py", line 6, in from pipelayer.pipeline import Pipeline # NOQA F401 File "C:\Github\Python\venv\lib\site-packages\pipelayer\pipeline.py", line 10, in from pipelayer.protocol import IFilter, IStep File "C:\Github\Python\venv\lib\site-packages\pipelayer\protocol.py", line 5, in from typing import (Any, Callable, List, Optional, Protocol, Tuple, ImportError: cannot import name 'Protocol' from 'typing' (C:\Program Files\Python\lib\typing.py)

    opened by zizu1985 3
  • Version 0.2.0

    Version 0.2.0

    • Adds support for module/static/lambda funcs as pipeline filters and pre/post process functions
    • Adds support type of Filter as a pipeline filter
    • Example project "simple_pipelayer" can run as a standalone app
    • Initializes/validates all filters/processors before running the pipeline
    opened by greater-than 0
Releases(v0.7.0)
Owner
greaterthan
greaterthan
A proof-of-concept CherryPy inspired Python micro framework

Varmkorv Varmkorv is a CherryPy inspired micro framework using Werkzeug. This is just a proof of concept. You are free to use it if you like, or find

Magnus Karlsson 1 Nov 22, 2021
Web3.py plugin for using Flashbots' bundle APIs

This library works by injecting a new module in the Web3.py instance, which allows submitting "bundles" of transactions directly to miners. This is do

Flashbots 293 Dec 31, 2022
Flask-Potion is a RESTful API framework for Flask and SQLAlchemy, Peewee or MongoEngine

Flask-Potion Description Flask-Potion is a powerful Flask extension for building RESTful JSON APIs. Potion features include validation, model resource

DTU Biosustain 491 Dec 08, 2022
Lemon is an async and lightweight API framework for python

Lemon is an async and lightweight API framework for python . Inspired by Koa and Sanic .

Joway 29 Nov 20, 2022
The lightning-fast ASGI server. ?

The lightning-fast ASGI server. Documentation: https://www.uvicorn.org Community: https://discuss.encode.io/c/uvicorn Requirements: Python 3.6+ (For P

Encode 6k Jan 03, 2023
Flask Sugar is a web framework for building APIs with Flask, Pydantic and Python 3.6+ type hints.

Flask Sugar is a web framework for building APIs with Flask, Pydantic and Python 3.6+ type hints. check parameters and generate API documents automatically. Flask Sugar是一个基于flask,pyddantic,类型注解的API框架

162 Dec 26, 2022
A shopping list and kitchen inventory management app.

Flask React Project This is the backend for the Flask React project. Getting started Clone this repository (only this branch) git clone https://github

11 Jun 03, 2022
Quiz Web App with Flask and MongoDB as the Databases

quiz-app Quiz Web Application made with flask and mongodb as the Databases Before you run this application, change the inside MONGODB_URI ( in config.

gibran abdillah 7 Dec 14, 2022
web.py is a web framework for python that is as simple as it is powerful.

web.py is a web framework for Python that is as simple as it is powerful. Visit http://webpy.org/ for more information. The latest stable release 0.62

5.8k Dec 30, 2022
Low code web framework for real world applications, in Python and Javascript

Full-stack web application framework that uses Python and MariaDB on the server side and a tightly integrated client side library.

Frappe 4.3k Dec 30, 2022
Persistent remote applications for X11; screen sharing for X11, MacOS and MSWindows.

Table of Contents About Installation Usage Help About Xpra is known as "screen for X" : its seamless mode allows you to run X11 programs, usually on a

xpra.org 785 Dec 30, 2022
PipeLayer is a lightweight Python pipeline framework

PipeLayer is a lightweight Python pipeline framework. Define a series of steps, and chain them together to create modular applications

greaterthan 64 Jul 21, 2022
Screaming-fast Python 3.5+ HTTP toolkit integrated with pipelining HTTP server based on uvloop and picohttpparser.

Japronto! There is no new project development happening at the moment, but it's not abandoned either. Pull requests and new maintainers are welcome. I

Paweł Piotr Przeradowski 8.6k Dec 29, 2022
Async Python 3.6+ web server/framework | Build fast. Run fast.

Sanic | Build fast. Run fast. Build Docs Package Support Stats Sanic is a Python 3.6+ web server and web framework that's written to go fast. It allow

Sanic Community Organization 16.7k Jan 08, 2023
Serverless Python

Zappa - Serverless Python About Installation and Configuration Running the Initial Setup / Settings Basic Usage Initial Deployments Updates Rollback S

Rich Jones 11.9k Jan 01, 2023
Phoenix LiveView but for Django

Reactor, a LiveView library for Django Reactor enables you to do something similar to Phoenix framework LiveView using Django Channels. What's in the

Eddy Ernesto del Valle Pino 526 Jan 02, 2023
Trame let you weave various components and technologies into a Web Application solely written in Python.

Trame Trame aims to be a framework for building interactive applications using a web front-end in plain Python. Such applications can be used locally

Kitware, Inc. 85 Dec 29, 2022
A minimal, extensible, fast and productive API framework for Python 3.

molten A minimal, extensible, fast and productive API framework for Python 3. Changelog: https://moltenframework.com/changelog.html Community: https:/

Bogdan Popa 980 Nov 28, 2022
News search API developed for the purposes of the ColdCase Project.

Saxion - Cold Case - News Search API Setup Local – Linux/MacOS Make sure you have python 3.9 and pip 21 installed. This project uses a MySQL database,

Dimitar Rangelov 3 Jul 01, 2021
The source code to the Midnight project

MidnightSniper Started: 24/08/2021 Ended: 24/10/2021 What? This is the source code to a project developed to snipe minecraft names Why release? The ad

Kami 2 Dec 03, 2021