Yet another Airflow plugin using CLI command as RESTful api, supports Airflow v2.X.

Overview

中文版文档

Airflow Extended API Plugin

Airflow Extended API, which export airflow CLI command as REST-ful API to extend the ability of airflow official API.

This plugin is available for airflow 2.x Version and extensible, as you can easily define your own API to execute any Airflow CLI command so that it fits your demand.

Current Supported Commands

The following commands are supported now, and more is coming.

  • airflow dags backfill
  • airflow tasks run
  • airflow tasks clear

Plugin Install

  1. Install the plugin via pip
  pip install airflow-extended-api
  1. Restart the Airflow Web Server

  2. Open Airflow UI in Docs - Extended API OpenAPI or http://localhost:8080/ to view extended API details in swagger UI. img.png

Usage

Examples

curl request example:

curl -X POST --user "airflow:airflow" https://localhost:8080/api/extended/clear -H "Content-Type: application/json" -d '{"dagName": "string","downstream": true,"endDate": "2019-08-24T14:15:22Z","jobName": "string","startDate": "2019-08-24T14:15:22Z","username": "Extended API"}'

Response Schema:

{
  "executed_command": "string",
  "exit_code": 0,
  "output_info": [
    "string"
  ],
  "error_info": [
    "string"
  ]
}

curl without Credentials data

Note that you will need to pass credentials' data in --user "{username}:{password}" format, or you will get an Unauthorized error.

curl -X POST http://127.0.0.1:8080/api/extended/clear -H "Content-Type: application/json" -d '{"dagName": "string","downstream": true,"endDate": "2019-08-24T14:15:22Z","jobName": "string","startDate": "2019-08-24T14:15:22Z","username": "Extended API"}'

response

{
  "detail": null,
  "status": 401,
  "title": "Unauthorized",
  "type": "https://airflow.apache.org/docs/apache-airflow/2.2.5/stable-rest-api-ref.html#section/Errors/Unauthenticated"
}

curl with wrong CLI Command

curl -X POST --user "airflow:airflow"  http://127.0.0.1:8080/api/extended/clear -H "Content-Type: application/json" -d '{"dagName": "string","downstream": true,"endDate": "2019-08-24T14:15:22Z","jobName": "string","startDate": "2019-08-24T14:15:22Z","username": "Extended API"}'

response

{
  "error_info": [
    "Traceback (most recent call last):",
    "  File \"/home/airflow/.local/bin/airflow\", line 8, in <module>",
    "    sys.exit(main())",
    "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/__main__.py\", line 48, in main",
    "    args.func(args)",
    "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py\", line 48, in command",
    "    return func(*args, **kwargs)",
    "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py\", line 92, in wrapper",
    "    return f(*args, **kwargs)",
    "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py\", line 506, in task_clear",
    "    dags = get_dags(args.subdir, args.dag_id, use_regex=args.dag_regex)",
    "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py\", line 203, in get_dags",
    "    return [get_dag(subdir, dag_id)]",
    "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py\", line 193, in get_dag",
    "    f\"Dag {dag_id!r} could not be found; either it does not exist or it failed to parse.\"",
    "airflow.exceptions.AirflowException: Dag 'string' could not be found; either it does not exist or it failed to parse.",
    ""
  ],
  "executed_command": "airflow tasks clear string -e 2019-08-24T14:15:22+00:00 -s 2019-08-24T14:15:22+00:00 -t string -y -d",
  "exit_code": 1,
  "output_info": [
    "[\u001b[34m2022-04-22 10:05:50,538\u001b[0m] {\u001b[34mdagbag.py:\u001b[0m500} INFO\u001b[0m - Filling up the DagBag from /opt/airflow/dags\u001b[0m",
    ""
  ]
}

Project Plan

  • support custom configuration

Links and References

You might also like...
A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!
A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!

Streamify A data pipeline with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more! Description Objective The project will stre

A python script to decrypt media files encrypted using the Android application 'Secret Calculator Photo Vault'. Supports brute force of PIN also.

A python script to decrypt media files encrypted using the Android application 'Secret Calculator Photo Vault'. Supports brute force of PIN also.

✨ Udemy Coupon Finder For Discord. Supports Turkish & English Language.

Udemy Course Finder Bot | Udemy Kupon Bulucu Botu This bot finds new udemy coupons and sends to the channel. Before Setup You must have python = 3.6

An esoteric programming language that supports concurrency, regex, and web requests.

The Hofstadter Esoteric Programming Language Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's La

Zeus is an open source flight intellingence tool which supports more than 13,000+ airlines and 250+ countries.

Zeus Zeus is an open source flight intellingence tool which supports more than 13,000+ airlines and 250+ countries. Any flight worldwide, at your fing

Aero is an open source airplane intelligence tool. Aero supports more than 13,000 airlines and 250 countries. Any flight worldwide at your fingertips.

Aero Aero supports more than 13,000 airlines and 250 countries. Any flight worldwide at your fingertips. Features Main : Flight lookup Aircraft lookup

A feed generator. Currently supports generating RSS feeds from Google, Bing, and Yahoo news.

A feed generator. Currently supports generating RSS feeds from Google, Bing, and Yahoo news.

Implements a polyglot REPL which supports multiple languages and shared meta-object protocol scope between REPLs.
Implements a polyglot REPL which supports multiple languages and shared meta-object protocol scope between REPLs.

MetaCall Polyglot REPL Description This repository implements a Polyglot REPL which shares the state of the meta-object protocol between the REPLs. Us

A minimalist personal blogging system that natively supports Markdown, LaTeX, and code highlighting.
A minimalist personal blogging system that natively supports Markdown, LaTeX, and code highlighting.

December Welcome to the December blogging system's code repository! Introduction December is a minimalist personal blogging system that natively suppo

Releases(v1.1.2)
  • v1.1.2(Apr 22, 2022)

    中文版文档

    Airflow Extended API Plugin

    Airflow Extended API, which export airflow CLI command as REST-ful API to extend the ability of airflow official API.

    Features

    • 👏Available: Probably the only available CLI command plugin on Github that supports the Airflow 2.x version.
    • 🎉Extensible: Easily define your own API to execute any Airflow CLI command so that it fits your demand.

    Plugin Install

    1. Install the plugin via pip
      pip install airflow-extended-api
    
    1. Restart the Airflow Web Server

    2. Open Airflow UI in Docs - Extended API OpenAPI or http://localhost:8080/ to view extended API details in swagger UI. img.png

    Usage

    Examples

    curl request example:

    curl -X POST --user "airflow:airflow" https://localhost:8080/api/extended/clear -H "Content-Type: application/json" -d '{"dagName": "string","downstream": true,"endDate": "2019-08-24T14:15:22Z","jobName": "string","startDate": "2019-08-24T14:15:22Z","username": "Extended API"}'
    

    Response Schema:

    {
      "executed_command": "string",
      "exit_code": 0,
      "output_info": [
        "string"
      ],
      "error_info": [
        "string"
      ]
    }
    

    curl without Credentials data

    Note that you will need to pass credentials' data in --user "{username}:{password}" format, or you will get an Unauthorized error.

    curl -X POST http://127.0.0.1:8080/api/extended/clear -H "Content-Type: application/json" -d '{"dagName": "string","downstream": true,"endDate": "2019-08-24T14:15:22Z","jobName": "string","startDate": "2019-08-24T14:15:22Z","username": "Extended API"}'
    

    response

    {
      "detail": null,
      "status": 401,
      "title": "Unauthorized",
      "type": "https://airflow.apache.org/docs/apache-airflow/2.2.5/stable-rest-api-ref.html#section/Errors/Unauthenticated"
    }
    

    curl with wrong CLI Command

    curl -X POST --user "airflow:airflow"  http://127.0.0.1:8080/api/extended/clear -H "Content-Type: application/json" -d '{"dagName": "string","downstream": true,"endDate": "2019-08-24T14:15:22Z","jobName": "string","startDate": "2019-08-24T14:15:22Z","username": "Extended API"}'
    

    response

    {
      "error_info": [
        "Traceback (most recent call last):",
        "  File \"/home/airflow/.local/bin/airflow\", line 8, in <module>",
        "    sys.exit(main())",
        "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/__main__.py\", line 48, in main",
        "    args.func(args)",
        "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py\", line 48, in command",
        "    return func(*args, **kwargs)",
        "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py\", line 92, in wrapper",
        "    return f(*args, **kwargs)",
        "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py\", line 506, in task_clear",
        "    dags = get_dags(args.subdir, args.dag_id, use_regex=args.dag_regex)",
        "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py\", line 203, in get_dags",
        "    return [get_dag(subdir, dag_id)]",
        "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py\", line 193, in get_dag",
        "    f\"Dag {dag_id!r} could not be found; either it does not exist or it failed to parse.\"",
        "airflow.exceptions.AirflowException: Dag 'string' could not be found; either it does not exist or it failed to parse.",
        ""
      ],
      "executed_command": "airflow tasks clear string -e 2019-08-24T14:15:22+00:00 -s 2019-08-24T14:15:22+00:00 -t string -y -d",
      "exit_code": 1,
      "output_info": [
        "[\u001b[34m2022-04-22 10:05:50,538\u001b[0m] {\u001b[34mdagbag.py:\u001b[0m500} INFO\u001b[0m - Filling up the DagBag from /opt/airflow/dags\u001b[0m",
        ""
      ]
    }
    

    Project Plan

    • [ ] support dags backfill cli command
    • [ ] support custom configuration

    Links and References

    Source code(tar.gz)
    Source code(zip)
Owner
Eric Cao
Stay hungry, Stay foolish。
Eric Cao
OWASP Foundation Web Respository

WWWGrep OWASP Foundation Web Respository Author: Mark Deen & Aditi Mohan Introduction WWWGrep is a rapid search “grepping” mechanism that examines HTM

OWASP 34 Jun 15, 2022
A PDM plugin to publish to PyPI

PDM Publish A PDM plugin to publish to PyPI NOTE: Consider if you need this over using twine directly Installation If you installed pdm via pipx: pipx

Branch Vincent 20 Aug 06, 2022
A funny alarm clock I made in python

Wacky-Alarm-Clock Basically, I kept forgetting to take my medications, so I thought it would be a fun project to code my own alarm clock and make it r

1 Nov 18, 2021
An integrated library for checking email if it is registered on social media

An integrated library for checking email if it is registered on social media

Sidra ELEzz 13 Dec 08, 2022
Open HW & SW for Scanning Electron Microscopes

OpenSEM Project Status: Preliminary The purpose of this project is to create a modern and open-source hardware and software platform for using vintage

Steven Lovegrove 7 Nov 01, 2022
A Python version of Canvacord

A copy of canvacord made in python! Installation Run any of these commands in terminal: Mac / Linux pip install canvacord Windows python -m pip insta

10 Mar 28, 2022
Scientific color maps and standardization tools

Scicomap is a package that provides scientific color maps and tools to standardize your favourite color maps if you don't like the built-in ones. Scicomap currently provides sequential, bi-sequential

Thomas Bury 14 Nov 30, 2022
SimilarWeb for Team ACT v.0.0.1

SimilarWeb for Team ACT v.0.0.1 This module has been built to provide a better environment specifically for Similarweb in Team ACT. This module itself

Sunkyeong Lee 0 Dec 29, 2021
Repository, with small useful and functional applications

Repositorio,com pequenos aplicativos uteis e funcionais A ideia e ir deselvolvendo pequenos aplicativos funcionais e adicionar a este repositorio List

GabrielDuke 6 Dec 06, 2021
Set up a sidechain for the XRPL quickly and easily

Sidechain Launch Kit Introduction This directory contains python scripts to tests and explore side chains. This document walks through the steps to se

Xpring Engineering 15 Dec 08, 2022
FBChecker Account using python , package requests and web old facebook

fbcek FBChecker Account using python , package requests and web old facebook using python 3.x apt upgrade -y apt update -y pkg install bash -y pkg ins

XnuxersXploitXen 5 Dec 24, 2022
Create or join a private chatroom without any third-party middlemen in less than 30 seconds, available through an AES encrypted password protected link.

PY-CHAT Create or join a private chatroom without any third-party middlemen in less than 30 seconds, available through an AES encrypted password prote

1 Nov 24, 2021
Tutor plugin for integration of Open edX with a Richie course catalog

Richie plugin for Tutor This is a plugin to integrate Richie, the learning portal CMS, with Open edX. The integration takes the form of a Tutor plugin

Overhang.IO 2 Sep 08, 2022
Small C-like language compiler for the Uxn assembly language

Pyuxncle is a single-pass compiler for a small subset of C (albeit without the std library). This compiler targets Uxntal, the assembly language of the Uxn virtual computer. The output Uxntal is not

CPunch 13 Jun 28, 2022
An implementation of Ray Tracing in One Weekend using Taichi

又一个Taichi语言的Ray Tracer 背景简介 这个Ray Tracer基本上是照搬了Peter Shirley的第一本小书Ray Tracing in One Weekend,在我写的时候参考的是Version 3.2.3这个版本。应该比其他中文博客删改了不少内容。果然Peter Shir

张皓 30 Nov 21, 2022
Open source home automation that puts local control and privacy first

Home Assistant Open source home automation that puts local control and privacy first. Powered by a worldwide community of tinkerers and DIY enthusiast

Home Assistant 57k Jan 02, 2023
Hy - A dialect of Lisp that's embedded in Python

Hy Lisp and Python should love each other. Let's make it happen. Hy is a Lisp dialect that's embedded in Python. Since Hy transforms its Lisp code int

Hy Society 4.4k Jan 02, 2023
Gerador do Arquivo Magnético Sintegra em Python

pysintegra é uma lib simples com o objetivo de facilitar a geração do arquivo SINTEGRA seguindo o Convênio ICMS 57/95. Com o surgimento do SPED, muito

Felipe Correa 5 Apr 07, 2022
Demo repository for Saltconf21 talk - Testing strategies for Salt states

Saltconf21 testing strategies Demonstration repository for my Saltconf21 talk "Strategies for testing Salt states" Talk recording Slides and demos Get

Barney Sowood 3 Mar 31, 2022
This is a far more in-depth and advanced version of "Write user interface to a file API Sample"

Fusion360-Write-UserInterface This is a far more in-depth and advanced version of "Write user interface to a file API Sample" from https://help.autode

4 Mar 18, 2022