Pokemon catch events project to demonstrate data pipeline on AWS

Overview

Pokemon Catches Data Pipeline

  • This is a sample project to practice end-to-end data project;
  • Terraform is used to deploy infrastructure;
  • Kafka is the tool used to receive pokemon catches events;
  • Target Database is Elasticsearch;
  • Data visualization tool is Kibana.

Disclaimer

  • This project is not considering the best security practices;
  • You can use this project to practice and learning how to connect the tools but not for prodution;
  • In order to run this idea in production, please consider adding more safety configurition on each service;
  • When you run terraform, it will create entire infrastructure and events will start to be generated;
  • You can go ahead and open up Elasticsearch, create an index pattern and add some Kibana visualization;
  • Enjoy and have fun while learning!

Alt text

Requirements

  • terraform
  • aws cli version 2
  • python3

Manual Configuration

  • Set up ~./credentials file
  • export AWS_PROFILE=<your_profile>
  • export AWS_REGION=<your_region>

Setup

git clone [email protected]:vitorcarra/pokemon-catch-pipeline.git

Terraform Deploy

  • Rename terraform.tfvars.template to terraform.tfvars
  • Modify terraform.tfvars with your values
terraform init
terraform validate
terraform plan -var-file terraform.tfvars
terraform apply -var-file terraform.tfvars

Terraform

Terraform

Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. Terraform can manage existing and popular service providers as well as custom in-house solutions.

Owner
Vitor Carra
I work with data technologies and cloud computing. I'm open to discuss any tech topic, including Python, AWS and Data.
Vitor Carra
Python library and cli util for https://www.zerochan.net/

Zerochan Library for Zerochan.net with pics parsing and downloader included! Features CLI utility for pics downloading from zerochan.net Library for c

kiriharu 10 Oct 11, 2022
Python Projects is an Open Source to enhance your python skills

Welcome! 👋🏽 Python Project is Open Source to enhance your python skills. You're free to contribute. 🤓 You just need to give us your scripts written

Tristán 6 Nov 28, 2022
A simple watcher for the XTZ/kUSD pool on Quipuswap

Kolibri Quipuswap Watcher This repo holds the source code for the QuipuBot bot deployed to the #quipuswap-updates channel in the Kolibri Discord Setup

Hover Labs 1 Nov 18, 2021
Mpis-ex7 - Implementation of tasks 1, 2, 3 for Metody Probabilistyczne i Statystyka Lista 7

Implementations of task 1, 2 and 3 from here Author: Maciej Bazela Index: 261743 Each task was implemented in Python 3. I've used Cython to speed up e

Maciej Bazela 1 Feb 27, 2022
Python plugin/extra to load data files from an external source (such as AWS S3) to a local directory

Data Loader Plugin - Python Table of Content (ToC) Data Loader Plugin - Python Table of Content (ToC) Overview References Python module Python virtual

Cloud Helpers 2 Jan 10, 2022
Python script which allows for automatic registration in Golfbox

Python script which allows for automatic registration in Golfbox

Guðni Þór Björnsson 8 Dec 04, 2021
A little tool that uses LLVM to extract simple "what does this do" level instruction information from all architectures.

moirai: MOre InstRuctions and Information Backcronym. Anyway, this is a small project to extract useful instruction definitions from LLVM's platform d

2 Jul 30, 2022
Python scripts to interact with Upper Deck ePack online trading card platform

This script should connect to the Upper Deck ePack API using your browser cookies and download a list of your current collection and save it as a CSV.

Adrian Kent 1 Nov 22, 2021
a simple functional programming language compiler written in python

Functional Programming Language A compiler for my small functional language. Written in python with SLY lexer/parser generator library. Requirements p

Ashkan Laei 3 Nov 05, 2021
Shell scripts made simple 🐚

zxpy Shell scripts made simple 🐚 Inspired by Google's zx, but made much simpler and more accessible using Python. Rationale Bash is cool, and it's ex

Tushar Sadhwani 492 Dec 27, 2022
A simple desktop application to scan and export Genshin Impact Artifacts.

「天目」 -- Amenoma 简体中文 | English 「天目流的诀窍就是滴水穿石的耐心和全力以赴的意志」 扫描背包中的圣遗物,并导出至 json 格式。之后可导入圣遗物分析工具( 莫娜占卜铺 、 MingyuLab 、 Genshin Optimizer 进行计算与规划等。 已支持 原神2.

夏至 475 Dec 30, 2022
To lazy to read your homework ? Get it done with LOL

LOL To lazy to read your homework ? Get it done with LOL Needs python 3.x L:::::::::L OO:::::::::OO L:::::::::L L:::::::

KorryKatti 4 Dec 08, 2022
MySQL Connectivity based project. Contains various functions of a Store-Management-System

An Intermediate Level Python - MySQL Connectivity based project. Contains various functions of a Store-Management-System.

Yash Wadhvani 2 Nov 21, 2022
A repository containing an introduction to Panel made to be support videos and talks.

👍 Awesome Panel - Introduction to Panel THIS REPO IS WORK IN PROGRESS. PRE-ALPHA Panel is a very powerful framework for exploratory data analysis and

Marc Skov Madsen 51 Nov 17, 2022
Install Firefox from Mozilla.org easily, complete with .desktop file creation.

firefox-installer Install Firefox from Mozilla.org easily, complete with .desktop file creation. Dependencies Python 3 Python LXML Debian/Ubuntu: sudo

rany 7 Nov 04, 2022
This tool don't used illegal ativity

ETHICALTOOL This tool for only educational purposes don't used illegal ativity @onlinehacking this tool for pkg update && pkg upgrade && pkg install g

Mrkarthick 4 Dec 23, 2021
SQL centered, docker process running game

REQUIREMENTS Linux Docker Python/bash set up image "docker build -t game ." create db container "run my_whatever/game_docker/pdb create" # creating po

1 Jan 11, 2022
🙌Kart of 210+ projects based on machine learning, deep learning, computer vision, natural language processing and all. Show your support by ✨ this repository.

ML-ProjectKart 📌 Repository This kart showcases the finest collection of all projects based on machine learning, deep learning, computer vision, natu

Prathima Kadari 203 Dec 28, 2022
Random Turkish name generator with realistic probabilities.

trnames Random Turkish name generator with realistic probabilities. Based on Trey Hunner's names package. Installation The package can be installed us

Kaan Öztürk 20 Jan 02, 2023
Url-check-migration-python - A python script using Apica API's to migrate URL checks between environments

url-check-migration-python A python script using Apica API's to migrate URL chec

Angelo Aquino 1 Feb 16, 2022