Pokemon catch events project to demonstrate data pipeline on AWS

Overview

Pokemon Catches Data Pipeline

  • This is a sample project to practice end-to-end data project;
  • Terraform is used to deploy infrastructure;
  • Kafka is the tool used to receive pokemon catches events;
  • Target Database is Elasticsearch;
  • Data visualization tool is Kibana.

Disclaimer

  • This project is not considering the best security practices;
  • You can use this project to practice and learning how to connect the tools but not for prodution;
  • In order to run this idea in production, please consider adding more safety configurition on each service;
  • When you run terraform, it will create entire infrastructure and events will start to be generated;
  • You can go ahead and open up Elasticsearch, create an index pattern and add some Kibana visualization;
  • Enjoy and have fun while learning!

Alt text

Requirements

  • terraform
  • aws cli version 2
  • python3

Manual Configuration

  • Set up ~./credentials file
  • export AWS_PROFILE=<your_profile>
  • export AWS_REGION=<your_region>

Setup

git clone [email protected]:vitorcarra/pokemon-catch-pipeline.git

Terraform Deploy

  • Rename terraform.tfvars.template to terraform.tfvars
  • Modify terraform.tfvars with your values
terraform init
terraform validate
terraform plan -var-file terraform.tfvars
terraform apply -var-file terraform.tfvars

Terraform

Terraform

Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. Terraform can manage existing and popular service providers as well as custom in-house solutions.

Owner
Vitor Carra
I work with data technologies and cloud computing. I'm open to discuss any tech topic, including Python, AWS and Data.
Vitor Carra
Lightweight and Modern kernel for VK Bots

This is the kernel for creating VK Bots written in Python 3.9

Yrvijo 4 Nov 21, 2021
Singularity Containers on Apple M1 (ARM64)

Singularity Containers on Apple M1 (ARM64) This is a repository containing a ready-to-use environment for singularity in arm64 (M1). It has been prepa

Manuel Parra 4 Nov 14, 2022
An implementation of multimap with per-item expiration backed up by Redis.

MultiMapWithTTL An implementation of multimap with per-item expiration backed up by Redis. Documentation: https://loggi.github.io/python-multimapwitht

Loggi 2 Jan 17, 2022
Python’s bokeh, holoviews, matplotlib, plotly, seaborn package-based visualizations about COVID statistics eventually hosted as a web app on Heroku

COVID-Watch-NYC-Python-Visualization-App Python’s bokeh, holoviews, matplotlib, plotly, seaborn package-based visualizations about COVID statistics ev

Aarif Munwar Jahan 1 Jan 04, 2022
Demo code for "Logs in distributed systems" webinar

Hexlet Logs Demo Пререквизиты docker-compose python3 Учетка в DataDog Базовое понимание, что такое логи (можно почитать гайд

Anton Markelov 1 Dec 01, 2021
My Solutions to 120 commonly asked data science interview questions.

Data_Science_Interview_Questions Introduction 👋 Here are the answers to 120 Data Science Interview Questions The above answer some is modified based

Milaan Parmar / Милан пармар / _米兰 帕尔马 181 Dec 31, 2022
Python module used to generate random facts

Randfacts is a python library that generates random facts. You can use randfacts.get_fact() to return a random fun fact. Disclaimer: Facts are not gua

Tabulate 14 Dec 14, 2022
Nook is a simple, concatenative programming language written in Python.

Nook Nook is a simple, concatenative programming language written in Python. Status Nook is currently WIP. It lacks a lot of basic feature, and will n

Wumi4 4 Jul 20, 2022
freeCodeCamp Scientific Computing with Python Project for Certification.

Time_Calculator_freeCodeCamp freeCodeCamp Scientific Computing with Python Project for Certification. Write a function named add_time that takes in tw

Rajdeep Mondal 1 Dec 23, 2021
Meliodas Official 1.4 BombSquad Server Scripts

Noxious-Official-1.4-BombSquad-Server-Scripts Scripts Are Provided By Sparxtn Somewhat Edited By Me Scripts are Working Fine Just Download & Use It Be

Meliodas♡ 2 Oct 16, 2022
Alfred 4 Workflow to search through your maintained/watched/starred GitHub repositories.

Alfred 4 Workflow to search through your maintained/watched/starred GitHub repositories. Setup This workflow requires a number of Python modules. Thes

Bᴇʀɴᴅ Sᴄʜᴏʀɢᴇʀs 1 Oct 14, 2022
It's a repo for Cramer's rule, which is some math crap or something idk

It's a repo for Cramer's rule, which is some math crap or something idk (just a joke, it's not crap; don't take that seriously, math teachers)

Module64 0 Aug 31, 2022
A python library with various gambling and gaming classes

gamble is a simple library that implements a collection of some common gambling-related classes Features die, dice, d-notation cards, decks, hands pok

Jacobi Petrucciani 16 May 24, 2022
IPO Checker for NEPSE

IPO Checker Checks more than one account for an IPO. Usage: ipo_checker.py [-h] --file FILE IPO Checker for a list. optional arguments: -h, --help

Sagar Tamang 4 Sep 20, 2022
This is the course repository for the Spring 2022 iteration of MACS 30123 "Large-Scale Computing for the Social Sciences" at the University of Chicago.

Large-Scale Computing for the Social Sciences Spring 2022 - MACS 30123/MAPS 30123/PLSC 30123 Instructor Information TA Information TA Information Cour

6 May 06, 2022
A python script that changes your desktop background based on current weather and time of the day.

Desktop background wallpaper, based on current weather and time A python script that changes your computer's desktop background based on current weath

Maj Gaberšček 1 Nov 16, 2021
In this project, we are going to display the battery notification and the time left for the battery to drain out using the battery capacity value.

In this project, we are going to display the battery notification and the time left for the battery to drain out using the battery capacity value.

Ritoban Biswas 1 Dec 20, 2021
An html wrapper for python

MessySoup What is it? MessySoup is a python wrapper for html elements. While still a ways away, the main goal is to be able to build a wesbite straigh

4 Jan 05, 2022
Block when attacker want to bypass the limit of request

Block when attacker want to bypass the limit of request

iFanpS 1 Dec 01, 2021
A Python version of Canvacord

A copy of canvacord made in python! Table of contents Installation Examples Creating Images Links Downloads Installation Run any of these commands in

10 Mar 28, 2022