Streaming Finance Data with AWS Lambda

Overview

Streaming Finance Data with AWS Lambda

In this project, we will demonstrate a streaming pipeline to stream near real time stock price data using yfinance module.

The pipeline is broken down as follows:

  1. Data Transformer

    Stock price data is ingested by yfinance module based on a AWS lambda function running a python script. The program collects four data points - stock ticker, timestamp, high price for ticker, low price for ticket at five minute intervals. Output of this step leads to pushing of ingested data to a AWS Kinesis stream.

  2. Data Collector

    Kinesis stream collects the data from the lambda function and stores data files in designated AWS S3 buckets. Output of this step leads to S3 buckets full of data ready to be analyzed.

  3. Data Analyzer

    AWS Glue Crawler is configured to convert the data in the S3 bucket to database schema and tables. AWS Athena is then used to query the data using SQL and export resulting tables in .csv format. This particular query gets the highest hourly stock “high” per company. Output of this step creates csv tables that can be inputs to jupyter notebook files for further analysis

  4. Data Visualization

    Jupyter notebook is used to perform further analysis on the queried data from Athena. Data visualizations are designed to explore the data; the visualizations are as follows:

    a. Interactive Stock Trend Line Plot - Line plot to show stock price changes trend over time of day - single select panel widget buttons to allow for interactive stock ticker selection.

    b. Interactive Stock Trend Line Plot Cumulative - Line plot to show stock price changes trend for all stock tickers over time of day - multi select panel widget buttons to allow adding/removing certain stock tickers from the chart.

    c. Hourly Stock Heatmap - Heatmap to show stock prices distribution for all stock tickers over time of day - multi select panel widget buttons to allow adding/removing certain stock tickers from the chart.

    Finally, for easier access to this visualization module, a public web application is deployed that contains all of the three charts in this module and allows the user to access these interative charts via a web browser without needing to interact with the python backend. Here is the link to the web application: https://stockpriceviz.herokuapp.com/ https://stockpriceviz.herokuapp.com/

Running the program

  1. Create a AWS Kinesis stream that has an AWS S3 bucket destination
  2. Write a AWS Lambda function with python code in the "data_transformer.py" file (make sure to provide Kinesis permissions to the Lambda function)
  3. Configure a AWS Glue crawler on the S3 directory to create databae schema and tables
  4. Run query using the "query.sql" file on AWS Athena to get the highest hourly stock “high” per company.
  5. Load the "Analysis.ipynb" file in Jupyter notebook to perform further analysis and visualization

AWS Kinesis Configuration

kinesis-configuration

-- Aarif M Jahan -- Dec 17, 2021

Owner
Aarif Munwar Jahan
Aarif Munwar Jahan
Discord Selfbot, 90+ commands

Setting the bot up. STEP 1: copy the directory yook.club selfbot was downloaded and extracted into, open cmd and type "cd " then paste. STEP 2: python

yook 1 Dec 12, 2021
OpenQuake's Engine for Seismic Hazard and Risk Analysis

OpenQuake Engine The OpenQuake Engine is an open source application that allows users to compute seismic hazard and seismic risk of earthquakes on a g

Global Earthquake Model 281 Dec 21, 2022
A free and open-source SMS/Call bombing application

TBOMB V0.1 A free and open-source SMS/Call bombing application NOTE: For Termux To use the bomber type the following commands in Termux: pkg install g

ᴀɴᴋɪᴛ ᴋᴜᴍᴀʀ 2 Dec 07, 2021
A Python wrapper for the Dogehouse API.

Python wrapper for the dogehouse API Installation pip install dogehouse Example from dogehouse import DogeClient, event, command from dogehouse.entiti

Arthur 36 Jun 15, 2022
🦊 Powerfull Discord Nitro Generator

🦊 Follow me here 🦊 Discord | YouTube | Github ☕ Usage 💻 Downloading git clone https://github.com/KanekiWeb/Nitro-Generator/new/main pip insta

Kaneki 104 Jan 02, 2023
⚡ ʑɠ ცơɬ Is One Of The Fastest & Smoothest Bot On Telegram Based on Telethon ⚡

『ʑɠ ცơɬ』 ⚡ ʑɠ ცơɬ Is One Of The Fastest & Smoothest Bot On Telegram Based on Telethon ⚡ Status Of Bot Telegram 🏪 Dєρℓογ το нєяοκυ Variables APP_ID =

ʑɑʑɓɦɑɪ 0 Feb 12, 2022
ANKIT-OS/STYLISH-TEXT is a special repository. Its Is A Telegram Bot Which Can Translate Your Text Into 100+ Language

🔥 ᴳᴼᴼᴳᴸᴱ⁻ᵀᴿᴬᴺᔆᴸᴬᵀᴱᴿ 🔥 The owner would not be responsible for any kind of bans due to the bot. • ⚡ INSTALLING ⚡ • • 🛠️ Lᴀɴɢᴜᴀɢᴇs Aɴᴅ Tᴏᴏʟs 🔰 • If

ANKIT KUMAR 1 Dec 23, 2021
A bot written in python that send prefilled Google Forms. It supports multithreading for faster execution time.

GoogleFormsBot https://flassy.xyz https://github.com/Shawey/GoogleFormsBot Requirements: os (Default) ast (Default) threading (Default) configparser (

Shawey 1 Jul 10, 2022
Changes the Telegram bio, profile picture, first and last name to the song that the user is currently listening to.

TGBIOFY - Telegram & Spotify integration Changes the Telegram bio, profile picture, first and last name to the song that the user is currently listeni

elpideus 26 Dec 07, 2022
Backlog API v2 Client Library for Python

BacklogPy - Backlog API v2 Client Library for Python BacklogPy is Backlog API v2 Client Library for Python 2/3 Install You can install the client libr

Koudai Aono 7 Dec 16, 2022
Framework to make using Bottle less time-consuming and easier

A class for the Bottle API to reduce clutter and difficulty while creating a website.

Tygzy 0 Dec 26, 2022
This is a Anti Channel Ban Robots

AntiChannelBan This is a Anti Channel Ban Robots delete and ban message sent by channels Heroku Deployment 💜 Heroku is the best way to host ur Projec

BᵣₐyDₑₙ 25 Dec 10, 2021
SquireBot is a Discord bot designed to run and manage tournaments entirely within a Discord.

Overview SquireBot is a Discord bot designed to run and manage tournaments entirely within a Discord. The current intended usecase is Magic: the Gathe

7 Nov 29, 2022
BeeDrive: Open Source Privacy File Transfering System for Teams and Individual Developers

BeeDrive For privacy and convenience purposes, more and more people try to keep data on their own hardwires instead of third-party cloud services such

Xuansheng Wu 8 Oct 31, 2022
A management system designed for the employees of MIRAS (Art Gallery). It is used to sell/cancel tickets, book/cancel events and keeps track of all upcoming events.

Art-Galleria-Management-System Its a management system designed for the employees of MIRAS (Art Gallery). Backend : Python Frontend : Django Database

Areesha Tahir 8 Nov 30, 2022
A minimalist file manager for those who want to use Linux mobile devices.

Portfolio A minimalist file manager for those who want to use Linux mobile devices. Usage Tap to activate and press to select, to browse, open, copy,

Martin Abente Lahaye 71 Nov 18, 2022
GitHub Activity Generator - A script that helps you instantly generate a beautiful GitHub Contributions Graph for the last year.

GitHub Activity Generator A script that helps you instantly generate a beautiful GitHub Contributions Graph for the last year. Before 😐 😶 😒 After ?

1 Dec 30, 2021
WIOpy - Walmart Affiliate API Python wrapper

WalmartIO Python Wrapper - WIOpy A python wrapper for the Walmart io API. Only s

6 Nov 14, 2022
MSE5050/7050 Materials Informatics course at the University of Utah

MaterialsInformatics MSE5050/7050 Materials Informatics course at the University of Utah This github repo contains coursework content such as class sl

41 Dec 30, 2022
With this simple app you can customize your presence in Discord.

Discord Rich Presence This a simple console app which can customize your Discord Presence easily, with only one config file! Setup Go to Discord Devel

Mai 1 Jan 06, 2022