Project repository of Apache Airflow, deployed on Docker in Amazon EC2 via GitLab.

Overview

Airflow on Docker in EC2 + GitLab's CI/CD

Personal project for simple data pipeline using Airflow. Airflow will be installed inside Docker container, which will be deployed in Amazon's EC2. For continuous integration and continuous deployment (CI/CD), GitLab will be used.

Steps

  1. Set up EC2 server. Choose ubuntu. Download the ssh key (with .pem suffix). Open ports of the EC2 instance, especially port 8080.
  2. SSH inside your EC2 server. Install Docker and docker-compose.
  3. Create GitLab account. Create new repository on GitLab, and push this repository there.
  4. On your GitLab's project page, open Settings > CI/CD > Repository Variables. Configure several variables:
    • _AIRFLOW_WWW_USER_PASSWORD -> Arbitrary password for Airflow (Variable)
    • _AIRFLOW_WWW_USER_USERNAME -> Arbitrary username for Airflow (Variable)
    • EC2_ADDRESS -> IP address of your EC2 host (Variable)
    • GITLAB_PASSWORD -> GitLab password (Variable)
    • GITLAB_USERNAME -> GitLab username (Variable)
    • SSH_KEY_EC2 -> Your SSH key (with .pem suffix that you downloaded earlier) (File)
  5. Configure GitLab's runner for CI/CD
  6. Open gitlab-ci.yml, change line 25 and 26 with your email (that registered on GitLab) and Name. If your user name in EC2 is not default (ubuntu), change ubuntu in [email protected]_ADDRESS with the correct username
  7. Run CI/CD pipeline, check if the code is deployed properly.
  8. SSH to the server to inspect if anything goes wrong
  9. Open the Airflow UI in browser on EC2_IP_ADDRESS:8080
Owner
Ammar Chalifah
Comic artist and a biomedical engineering student from Indonesia.
Ammar Chalifah
App and Python library for parsing, writing, and validation of the STAND013 file format.

python-stand013 python-stand013 is a Python app and library for parsing, writing, and validation of the STAND013 file format. Features The following i

Oda 3 Nov 09, 2022
Cute study buddy that helps you study with the Pomodoro technique!

study-buddy Cute study buddy that helps you study with the Pomodoro (or Animedoro) technique! Kirby The Kirby folder has a Kirby, pink-themed Pomodoro

Ethan Emmanuel 1 Jan 19, 2022
A simple but complete exercise to learning Python

ResourceReservationProject This is a simple but complete exercise to learning Python. Task and flow chart We are going to do a new fork of the existin

2 Nov 14, 2022
Easily Generate Revolut Business Cards

RevBusinessCardGen Easily Generate Revolut Business Cards Prerequisites Before you begin, ensure you have met the following requirements: You have ins

Younes™ 35 Dec 14, 2022
Learn to code in any language. If

Learn to Code It is an intiiative undertaken by Student Ambassadors Club, Jamshoro for students who are absolute begineers in programming and want to

Student Ambassadors' Club at Mehran UET 15 Oct 19, 2022
WorldsCollide - Final Fantasy VI Randomizer

FFVI Worlds Collide Worlds Collide is an open worlds randomizer for Final Fantas

8 Jun 13, 2022
Mmr image postbot - Бот для создания изображений с новыми релизами в сообщество ВК MMR Aggregator

Mmr image postbot - Бот для создания изображений с новыми релизами в сообщество ВК MMR Aggregator

Max 3 Jan 07, 2022
A small project of two newbies, who wanted to learn something about Python language programming, via fun way.

HaveFun A small project of two newbies, who wanted to learn something about Python language programming, via fun way. What's this project about? Well.

Patryk Sobczak 2 Nov 24, 2021
A Tool to validate domestic New Zealand vaccine passes

Vaccine Validator Tool to validate domestic New Zealand vaccine passes Create a new virtual environment: python3 -m venv ./venv Activate virtual envi

8 May 01, 2022
The official Repository wherein newbies into Open Source can Contribute during the Hacktoberfest 2021

Hacktoberfest 2021 Get Started With your first Contrinution/Pull Request : Fork/Copy the repo by clicking the right most button on top of the page. Go

HacOkars 25 Aug 20, 2022
tox-gh is a tox plugin which helps running tox on GitHub Actions with multiple different Python versions on multiple workers in parallel

tox-gh is a tox plugin which helps running tox on GitHub Actions with multiple different Python versions on multiple workers in parallel. This project is inspired by tox-travis.

tox development team 19 Dec 26, 2022
"Cambio de monedas" Change-making problem with Python, dynamic programming best solutions,

Change-making-problem / Cambio de monedas Entendiendo el problema Dada una cantidad de dinero y una lista de denominaciones de monedas, encontrar el n

Juan Antonio Ayola Cortes 1 Dec 08, 2021
🔩 Like builtins, but boltons. 250+ constructs, recipes, and snippets which extend (and rely on nothing but) the Python standard library. Nothing like Michael Bolton.

Boltons boltons should be builtins. Boltons is a set of over 230 BSD-licensed, pure-Python utilities in the same spirit as — and yet conspicuously mis

Mahmoud Hashemi 6k Jan 06, 2023
Cash in on Expressed Barcode Tags (EBTs) from NGS Sequencing Data with Python

Cash in on Expressed Barcode Tags (EBTs) from NGS Sequencing Data with Python Cashier is a tool developed by Russell Durrett for the analysis and extr

3 Sep 11, 2022
Program Input Data Mahasiswa Oop

PROGRAM INPUT NILAI MAHASISWA MENGGUNAKAN OOP PENGERTIAN OOP object-oriented-programing/OOP adalah paradigma pemrograman berdasarkan konsep "objek", y

Maulana Reza Badrudin 1 Jan 05, 2022
The presented desktop application was made to solve 1d schrodinger eqation

schrodinger_equation_1d_solver The presented desktop application was made to solve 1d schrodinger eqation. It implements Numerov's algorithm (step by

Artem Kashapov 2 Dec 29, 2021
Source for the Fedora Silverblue and Kinoite variants.

Source for the Fedora Silverblue and Kinoite variants.

Fedora Kinoite 7 Aug 20, 2022
ThinkPHP全日志扫描工具,命令行版和BurpSuite插件版

ThinkPHP3和5日志扫描工具,提供命令行版和BurpSuite插件版,尽可能全的发掘网站日志信息 命令行版 安装 git clone https://github.com/r3change/TPLogScan.git cd TPLogScan/ pip install -r requireme

119 Dec 27, 2022
Pampy: The Pattern Matching for Python you always dreamed of.

Pampy: Pattern Matching for Python Pampy is pretty small (150 lines), reasonably fast, and often makes your code more readable and hence easier to rea

Claudio Santini 3.5k Dec 30, 2022