Project repository of Apache Airflow, deployed on Docker in Amazon EC2 via GitLab.

Overview

Airflow on Docker in EC2 + GitLab's CI/CD

Personal project for simple data pipeline using Airflow. Airflow will be installed inside Docker container, which will be deployed in Amazon's EC2. For continuous integration and continuous deployment (CI/CD), GitLab will be used.

Steps

  1. Set up EC2 server. Choose ubuntu. Download the ssh key (with .pem suffix). Open ports of the EC2 instance, especially port 8080.
  2. SSH inside your EC2 server. Install Docker and docker-compose.
  3. Create GitLab account. Create new repository on GitLab, and push this repository there.
  4. On your GitLab's project page, open Settings > CI/CD > Repository Variables. Configure several variables:
    • _AIRFLOW_WWW_USER_PASSWORD -> Arbitrary password for Airflow (Variable)
    • _AIRFLOW_WWW_USER_USERNAME -> Arbitrary username for Airflow (Variable)
    • EC2_ADDRESS -> IP address of your EC2 host (Variable)
    • GITLAB_PASSWORD -> GitLab password (Variable)
    • GITLAB_USERNAME -> GitLab username (Variable)
    • SSH_KEY_EC2 -> Your SSH key (with .pem suffix that you downloaded earlier) (File)
  5. Configure GitLab's runner for CI/CD
  6. Open gitlab-ci.yml, change line 25 and 26 with your email (that registered on GitLab) and Name. If your user name in EC2 is not default (ubuntu), change ubuntu in [email protected]_ADDRESS with the correct username
  7. Run CI/CD pipeline, check if the code is deployed properly.
  8. SSH to the server to inspect if anything goes wrong
  9. Open the Airflow UI in browser on EC2_IP_ADDRESS:8080
Owner
Ammar Chalifah
Comic artist and a biomedical engineering student from Indonesia.
Ammar Chalifah
A PowSyBl and Python integration based on GraalVM native image

PyPowSyBl The PyPowSyBl project gives access PowSyBl Java framework to Python developers. This Python integration relies on GraalVM to compile Java co

powsybl 23 Dec 14, 2022
A10 cipher - A Hill 2x2 cipher that totally gone wrong

A10_cipher This is a Hill 2x2 cipher that totally gone wrong, it encrypts with H

Caner Çetin 15 Oct 19, 2022
To check my COVID-19 vaccine appointment, I wrote an infinite loop that sends me a Whatsapp message hourly using Twilio and Selenium. It works on my Raspberry Pi computer.

COVID-19_vaccine_appointment To check my COVID-19 vaccine appointment, I wrote an infinite loop that sends me a Whatsapp message hourly using Twilio a

Ayyuce Demirbas 24 Dec 17, 2022
Basic-Killfeed - A simple DayZ Console Killfeed

Basic-Killfeed A simple DayZ Console Killfeed. Setup Install Python Version 3.10

Nick 1 Apr 25, 2022
Procscan is a quick and dirty python script used to look for potentially dangerous api call patterns in a Procmon PML file.

PROCSCAN Procscan is a quick and dirty python script used to look for potentially dangerous api call patterns in a Procmon PML file. Installation git

Daniel Santos 9 Sep 02, 2022
A basic layout of atm working of my local database

Software for working Banking service 😄 This project was developed for Banking service. mysql server is required To have mysql server on your system u

satya 1 Oct 21, 2021
Modern API wrapper for Genshin Impact built on asyncio and pydantic.

genshin.py Modern API wrapper for Genshin Impact built on asyncio and pydantic.

sadru 212 Jan 06, 2023
You'll learn about Iterators, Generators, Closure, Decorators, Property, and RegEx in detail with examples.

07_Python_Advanced_Topics Introduction 👋 In this tutorial, you will learn about: Python Iterators: They are objects that can be iterated upon. In thi

Milaan Parmar / Милан пармар / _米兰 帕尔马 252 Dec 23, 2022
HiSim - House Infrastructure Simulator

HiSim is a Python package for simulation and analysis of household scenarios using modern components as alternative to fossil fuel based ones.

FZJ-IEK3 17 Dec 17, 2022
A demo of a data science project using Kedro

iris Overview This is your new Kedro project, which was generated using Kedro 0.17.4. Take a look at the Kedro documentation to get started. Rules and

Khuyen Tran 14 Oct 14, 2022
Advanced IPv4 Subnet Calculator in Python3

Advanced IPv4 Subnet Calculator in Python3 Table of Contents Getting Started Installation How it works? SVI Configuration Template Previews Getting St

Osama Abbas 1 May 10, 2022
A Klipper plugin for accurate Z homing

Stable Z Homing for Klipper A Klipper plugin for accurate Z homing This plugin provides a new G-code command, STABLE_Z_HOME, which homes Z repeatedly

Matthew Lloyd 24 Dec 28, 2022
Domoticz-hyundai-kia - Domoticz Hyundai-Kia plugin for Domoticz home automation system

Domoticz Hyundai-Kia plugin Author: Creasol https://www.creasol.it/domotics For

Creasol 7 Aug 03, 2022
Create rangebased on lists or values of the range itself. Range any type. Can you imagine?

funcao-allrange-for-python3 Create rangebased on lists or values of the range itself. Range any type. Can you imagine? WARNING!!! THIS MODULE DID NOT

farioso-fernando 1 Feb 09, 2022
A small Blender addon for changing an object's local orientation while in edit mode

A small Blender addon for changing an object's local orientation while in edit mode.

Jonathan Lampel 50 Jan 06, 2023
3D Printed Flip Clock Design and Code

Smart Flip Clock 3D printed smart clock that puts a new twist on old technology. Making The Smart Flip Clock The first thing that must be done for thi

Thomas 105 Oct 17, 2022
Roblox Limited Sniper For Python

Info this is version 2.1 version 3 will support more options (install python: https://www.python.org) the program will buy any limited item with a pri

1 Dec 09, 2021
Vector tile server for the Wildfire Predictive Services Unit

wps-tileserver Vector tile server for the Wildfire Predictive Services Unit Overview The intention of this project is to: provide tools to easily spin

Province of British Columbia 6 Dec 20, 2022
AutoMetamon: Simple program to play Metamon automatically

AutoMetamon: Simple program to play Metamon automatically

Ngô Văn Tuấn 2 Sep 13, 2022