Yet another Airflow plugin using CLI command as RESTful api, supports Airflow v2.X.

Overview

中文版文档

Airflow Extended API Plugin

Airflow Extended API, which export airflow CLI command as REST-ful API to extend the ability of airflow official API.

This plugin is available for airflow 2.x Version and extensible, as you can easily define your own API to execute any Airflow CLI command so that it fits your demand.

Current Supported Commands

The following commands are supported now, and more is coming.

  • airflow dags backfill
  • airflow tasks run
  • airflow tasks clear

Plugin Install

  1. Install the plugin via pip
  pip install airflow-extended-api
  1. Restart the Airflow Web Server

  2. Open Airflow UI in Docs - Extended API OpenAPI or http://localhost:8080/ to view extended API details in swagger UI. img.png

Usage

Examples

curl request example:

curl -X POST --user "airflow:airflow" https://localhost:8080/api/extended/clear -H "Content-Type: application/json" -d '{"dagName": "string","downstream": true,"endDate": "2019-08-24T14:15:22Z","jobName": "string","startDate": "2019-08-24T14:15:22Z","username": "Extended API"}'

Response Schema:

{
  "executed_command": "string",
  "exit_code": 0,
  "output_info": [
    "string"
  ],
  "error_info": [
    "string"
  ]
}

curl without Credentials data

Note that you will need to pass credentials' data in --user "{username}:{password}" format, or you will get an Unauthorized error.

curl -X POST http://127.0.0.1:8080/api/extended/clear -H "Content-Type: application/json" -d '{"dagName": "string","downstream": true,"endDate": "2019-08-24T14:15:22Z","jobName": "string","startDate": "2019-08-24T14:15:22Z","username": "Extended API"}'

response

{
  "detail": null,
  "status": 401,
  "title": "Unauthorized",
  "type": "https://airflow.apache.org/docs/apache-airflow/2.2.5/stable-rest-api-ref.html#section/Errors/Unauthenticated"
}

curl with wrong CLI Command

curl -X POST --user "airflow:airflow"  http://127.0.0.1:8080/api/extended/clear -H "Content-Type: application/json" -d '{"dagName": "string","downstream": true,"endDate": "2019-08-24T14:15:22Z","jobName": "string","startDate": "2019-08-24T14:15:22Z","username": "Extended API"}'

response

{
  "error_info": [
    "Traceback (most recent call last):",
    "  File \"/home/airflow/.local/bin/airflow\", line 8, in <module>",
    "    sys.exit(main())",
    "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/__main__.py\", line 48, in main",
    "    args.func(args)",
    "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py\", line 48, in command",
    "    return func(*args, **kwargs)",
    "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py\", line 92, in wrapper",
    "    return f(*args, **kwargs)",
    "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py\", line 506, in task_clear",
    "    dags = get_dags(args.subdir, args.dag_id, use_regex=args.dag_regex)",
    "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py\", line 203, in get_dags",
    "    return [get_dag(subdir, dag_id)]",
    "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py\", line 193, in get_dag",
    "    f\"Dag {dag_id!r} could not be found; either it does not exist or it failed to parse.\"",
    "airflow.exceptions.AirflowException: Dag 'string' could not be found; either it does not exist or it failed to parse.",
    ""
  ],
  "executed_command": "airflow tasks clear string -e 2019-08-24T14:15:22+00:00 -s 2019-08-24T14:15:22+00:00 -t string -y -d",
  "exit_code": 1,
  "output_info": [
    "[\u001b[34m2022-04-22 10:05:50,538\u001b[0m] {\u001b[34mdagbag.py:\u001b[0m500} INFO\u001b[0m - Filling up the DagBag from /opt/airflow/dags\u001b[0m",
    ""
  ]
}

Project Plan

  • support custom configuration

Links and References

You might also like...
A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!
A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!

Streamify A data pipeline with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more! Description Objective The project will stre

A python script to decrypt media files encrypted using the Android application 'Secret Calculator Photo Vault'. Supports brute force of PIN also.

A python script to decrypt media files encrypted using the Android application 'Secret Calculator Photo Vault'. Supports brute force of PIN also.

✨ Udemy Coupon Finder For Discord. Supports Turkish & English Language.

Udemy Course Finder Bot | Udemy Kupon Bulucu Botu This bot finds new udemy coupons and sends to the channel. Before Setup You must have python = 3.6

An esoteric programming language that supports concurrency, regex, and web requests.

The Hofstadter Esoteric Programming Language Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's La

Zeus is an open source flight intellingence tool which supports more than 13,000+ airlines and 250+ countries.

Zeus Zeus is an open source flight intellingence tool which supports more than 13,000+ airlines and 250+ countries. Any flight worldwide, at your fing

Aero is an open source airplane intelligence tool. Aero supports more than 13,000 airlines and 250 countries. Any flight worldwide at your fingertips.

Aero Aero supports more than 13,000 airlines and 250 countries. Any flight worldwide at your fingertips. Features Main : Flight lookup Aircraft lookup

A feed generator. Currently supports generating RSS feeds from Google, Bing, and Yahoo news.

A feed generator. Currently supports generating RSS feeds from Google, Bing, and Yahoo news.

Implements a polyglot REPL which supports multiple languages and shared meta-object protocol scope between REPLs.
Implements a polyglot REPL which supports multiple languages and shared meta-object protocol scope between REPLs.

MetaCall Polyglot REPL Description This repository implements a Polyglot REPL which shares the state of the meta-object protocol between the REPLs. Us

A minimalist personal blogging system that natively supports Markdown, LaTeX, and code highlighting.
A minimalist personal blogging system that natively supports Markdown, LaTeX, and code highlighting.

December Welcome to the December blogging system's code repository! Introduction December is a minimalist personal blogging system that natively suppo

Releases(v1.1.2)
  • v1.1.2(Apr 22, 2022)

    中文版文档

    Airflow Extended API Plugin

    Airflow Extended API, which export airflow CLI command as REST-ful API to extend the ability of airflow official API.

    Features

    • 👏Available: Probably the only available CLI command plugin on Github that supports the Airflow 2.x version.
    • 🎉Extensible: Easily define your own API to execute any Airflow CLI command so that it fits your demand.

    Plugin Install

    1. Install the plugin via pip
      pip install airflow-extended-api
    
    1. Restart the Airflow Web Server

    2. Open Airflow UI in Docs - Extended API OpenAPI or http://localhost:8080/ to view extended API details in swagger UI. img.png

    Usage

    Examples

    curl request example:

    curl -X POST --user "airflow:airflow" https://localhost:8080/api/extended/clear -H "Content-Type: application/json" -d '{"dagName": "string","downstream": true,"endDate": "2019-08-24T14:15:22Z","jobName": "string","startDate": "2019-08-24T14:15:22Z","username": "Extended API"}'
    

    Response Schema:

    {
      "executed_command": "string",
      "exit_code": 0,
      "output_info": [
        "string"
      ],
      "error_info": [
        "string"
      ]
    }
    

    curl without Credentials data

    Note that you will need to pass credentials' data in --user "{username}:{password}" format, or you will get an Unauthorized error.

    curl -X POST http://127.0.0.1:8080/api/extended/clear -H "Content-Type: application/json" -d '{"dagName": "string","downstream": true,"endDate": "2019-08-24T14:15:22Z","jobName": "string","startDate": "2019-08-24T14:15:22Z","username": "Extended API"}'
    

    response

    {
      "detail": null,
      "status": 401,
      "title": "Unauthorized",
      "type": "https://airflow.apache.org/docs/apache-airflow/2.2.5/stable-rest-api-ref.html#section/Errors/Unauthenticated"
    }
    

    curl with wrong CLI Command

    curl -X POST --user "airflow:airflow"  http://127.0.0.1:8080/api/extended/clear -H "Content-Type: application/json" -d '{"dagName": "string","downstream": true,"endDate": "2019-08-24T14:15:22Z","jobName": "string","startDate": "2019-08-24T14:15:22Z","username": "Extended API"}'
    

    response

    {
      "error_info": [
        "Traceback (most recent call last):",
        "  File \"/home/airflow/.local/bin/airflow\", line 8, in <module>",
        "    sys.exit(main())",
        "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/__main__.py\", line 48, in main",
        "    args.func(args)",
        "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py\", line 48, in command",
        "    return func(*args, **kwargs)",
        "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py\", line 92, in wrapper",
        "    return f(*args, **kwargs)",
        "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py\", line 506, in task_clear",
        "    dags = get_dags(args.subdir, args.dag_id, use_regex=args.dag_regex)",
        "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py\", line 203, in get_dags",
        "    return [get_dag(subdir, dag_id)]",
        "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py\", line 193, in get_dag",
        "    f\"Dag {dag_id!r} could not be found; either it does not exist or it failed to parse.\"",
        "airflow.exceptions.AirflowException: Dag 'string' could not be found; either it does not exist or it failed to parse.",
        ""
      ],
      "executed_command": "airflow tasks clear string -e 2019-08-24T14:15:22+00:00 -s 2019-08-24T14:15:22+00:00 -t string -y -d",
      "exit_code": 1,
      "output_info": [
        "[\u001b[34m2022-04-22 10:05:50,538\u001b[0m] {\u001b[34mdagbag.py:\u001b[0m500} INFO\u001b[0m - Filling up the DagBag from /opt/airflow/dags\u001b[0m",
        ""
      ]
    }
    

    Project Plan

    • [ ] support dags backfill cli command
    • [ ] support custom configuration

    Links and References

    Source code(tar.gz)
    Source code(zip)
Owner
Eric Cao
Stay hungry, Stay foolish。
Eric Cao
Vaksina - Vaksina COVID QR Validation Checker With Python

Vaksina COVID QR Validation Checker Vaksina is a general purpose library intende

Michael Casadevall 33 Aug 20, 2022
Speed up your typing by some exercises in the multi-platform(Windows/Ubuntu).

Introduction This project purpose is speed up your typing by some exercises in the multi-platform(Windows/Ubuntu). Build Environment Software Environm

lyfer233 1 Mar 24, 2022
It is convenient to quickly import Python packages from the network.

It is convenient to quickly import Python packages from the network.

zmaplex 1 Jan 18, 2022
An application to see if your Ethereum staking validator(s) are members of the current or next post-Altair sync committees.

eth_sync_committee.py Since the Altair upgrade, 512 validators are randomly chosen every 256 epochs (~27 hours) to form a sync committee. Validators i

4 Oct 27, 2022
Gunakan Dengan Bijak!!

YMBF Made with ❤️ by ikiwzXD_ menu Results notice me: if you get cp results, save 3/7 days then log in. Install script on Termux $ pkg update && pkg u

Ikiwz 0 Jul 11, 2022
Companion Web site for Fluent Python, Second Edition

Fluent Python, the site Source code and content for fluentpython.com. The site complements Fluent Python, Second Edition with extra content that did n

Fluent Python 49 Dec 08, 2022
A silly RPG(Not MMO) made in python

Project_PyMMo A silly RPG(Not MMO) made in python, FOR WINDOWS 10 ONLY! Hello tester, to install pymmo follow the steps bellow: 1.First install python

0 Feb 08, 2022
We'll be using HTML, CSS and JavaScript for the frontend

We'll be using HTML, CSS and JavaScript for the frontend. Nothing to install in specific. Open your text-editor and start coding a beautiful front-end.

Mugada sai tilak 1 Dec 15, 2021
Fisherman is a free open source fishing bot written in python.

Fisherman is a free open source fishing bot written in python.

Pure | Cody 33 Jan 29, 2022
Simple Calculator Mobile Apps

Simple Calculator Mobile Apps Screenshoot If you want to try it please click the link below to download, this application is 100% safe no virus. link

0 Sep 24, 2022
Gerador de dafaces

🎴 DefaceGenerator Obs: esse script foi criado com a intenção de ajudar pessoas iniciantes no hacking que ainda não conseguem criar suas próprias defa

LordShinigami 3 Jan 09, 2022
Mixtaper - Web app to make mixtapes

Mixtaper A web app which allows you to input songs in the form of youtube links

suryansh 1 Feb 14, 2022
Create standalone, installable R Shiny apps using Electron

Create standalone, installable R Shiny apps using Electron

Chase Clark 5 Dec 24, 2021
Simple rofi script to choose player for playerctl to execute its command

rofi-playerctl-switcher simple rofi script to choose player for playerctl to execute its command Usage copy playerSwitch.py and playerctl.sh to ~/.con

2 Jan 03, 2022
Notifies server owners of mod updates, also notifies of player deaths and player joins through Discord.

ProjectZomboid-ServerAssistant Notifies server owners of mod updates, also notifies of player deaths and player joins through Discord. A Python based

3 Sep 30, 2022
Media Cloud Outlet Filtering

Using ABYZ and Media-Bias Fact-Check outlet databases, I've provided outlet CSV files for both and scripts to intended to match Media Cloud files to respective outlets.

Stephen Scarano 1 Feb 02, 2022
Random Programming Language Project

Crastle Random Programming Language Project Freedom of expression Are you a fan of curly brace languages? Then use curly braces! Not a fan of curly br

DevNugget 2 Dec 23, 2021
Software for visualization of RTStruct structures on CT images

This script is responsible for the operation of the program, it is responsible for both creating the GUI and the process of processing images from dicom files. The program is based on the use of the

Adam Piszczek 0 Jun 29, 2022
This is a Python script to detect rapid upwards price changes (pumps) in a cryptocurrency pairing

A python script to detect a rapid upwards price brekout (pump) in a cryptocurrency pairing, through pandas and Binance API.

3 May 25, 2022
Python library for parsing Godot scene files

Godot Parser This is a python library for parsing Godot scene (.tscn) and resource (.tres) files. It's intended to make it easier to automate certain

Steven Arcangeli 30 Jan 04, 2023