Streaming Finance Data with AWS Lambda

Overview

Streaming Finance Data with AWS Lambda

In this project, we will demonstrate a streaming pipeline to stream near real time stock price data using yfinance module.

The pipeline is broken down as follows:

  1. Data Transformer

    Stock price data is ingested by yfinance module based on a AWS lambda function running a python script. The program collects four data points - stock ticker, timestamp, high price for ticker, low price for ticket at five minute intervals. Output of this step leads to pushing of ingested data to a AWS Kinesis stream.

  2. Data Collector

    Kinesis stream collects the data from the lambda function and stores data files in designated AWS S3 buckets. Output of this step leads to S3 buckets full of data ready to be analyzed.

  3. Data Analyzer

    AWS Glue Crawler is configured to convert the data in the S3 bucket to database schema and tables. AWS Athena is then used to query the data using SQL and export resulting tables in .csv format. This particular query gets the highest hourly stock “high” per company. Output of this step creates csv tables that can be inputs to jupyter notebook files for further analysis

  4. Data Visualization

    Jupyter notebook is used to perform further analysis on the queried data from Athena. Data visualizations are designed to explore the data; the visualizations are as follows:

    a. Interactive Stock Trend Line Plot - Line plot to show stock price changes trend over time of day - single select panel widget buttons to allow for interactive stock ticker selection.

    b. Interactive Stock Trend Line Plot Cumulative - Line plot to show stock price changes trend for all stock tickers over time of day - multi select panel widget buttons to allow adding/removing certain stock tickers from the chart.

    c. Hourly Stock Heatmap - Heatmap to show stock prices distribution for all stock tickers over time of day - multi select panel widget buttons to allow adding/removing certain stock tickers from the chart.

    Finally, for easier access to this visualization module, a public web application is deployed that contains all of the three charts in this module and allows the user to access these interative charts via a web browser without needing to interact with the python backend. Here is the link to the web application: https://stockpriceviz.herokuapp.com/ https://stockpriceviz.herokuapp.com/

Running the program

  1. Create a AWS Kinesis stream that has an AWS S3 bucket destination
  2. Write a AWS Lambda function with python code in the "data_transformer.py" file (make sure to provide Kinesis permissions to the Lambda function)
  3. Configure a AWS Glue crawler on the S3 directory to create databae schema and tables
  4. Run query using the "query.sql" file on AWS Athena to get the highest hourly stock “high” per company.
  5. Load the "Analysis.ipynb" file in Jupyter notebook to perform further analysis and visualization

AWS Kinesis Configuration

kinesis-configuration

-- Aarif M Jahan -- Dec 17, 2021

Owner
Aarif Munwar Jahan
Aarif Munwar Jahan
A BOT TO FIND ID OF A STICKER.

sticker id A BOT TO FIND ID OF A STICKER. THIS REPOSITORY HAVE TWO BRANCHES FOR DEPLOY WITH COMMAND & WITHOUT COMMAND. Mandatory variables API_ID - Ge

Ashik Muhammed 3 Dec 29, 2022
A Multi-Tool with 30+Options.

A Multi-Tool with 30+Options.

Mervin404 15 Apr 12, 2022
NoChannelBot - Bot bans users, that send messages like channels

No Channel Bot Say "STOP" to users who send messages as channels! Bot prevents u

Andrew 10 Oct 05, 2022
"zpool iostats" for humans; find the slow parts of your ZFS pool

Getting the gist of zfs statistics vpool-demo.mp4 The ZFS command "zpool iostat" provides a histogram listing of how often it takes to do things in pa

Chad 57 Oct 24, 2022
Auto file forward bot with python

Auto-File-Forward-Bot Auto file forward bot. Without Admin Permission in FROM_CHANNEL Only Give Permission In your Telegram Personal Channel Please fo

Milas 1 Oct 15, 2021
Instadev - Crack Instagram IqbalDev

Crack Instagram IqbalDev ⇨ Install Script Di Termux $ pkg update && upgrade $

Dicky Wahyudi 1 Feb 27, 2022
Simple progressbar for discord

⚙️ DiscordProgressbar 📂 Установка | Installation pip install discordbar 📚 Документация | Documentation 📞 Связаться со мной | Сontact with me 📜 Ли

DenyS 26 Nov 30, 2022
Discord rich-presence implementation for VALORANT

not working on v1 anymore in favor of v2, but if there's any big bugs i'll try to fix them valorant-rich-presence-client Discord rich presence extensi

colinh 278 Jan 08, 2023
A simple Telegram bot that analyses a given word, built with python-telegram-bot

Telegram Word Bot A simple Telegram bot that analyses a given word, built with python-telegram-bot. The bot is fairly useless - unless you often need

Chris N 2 Jul 14, 2022
NFT Generator - A NFT Generator created using Python

NFT_Generator v1 An NFT Generator created using Python. This NFT Generation tool

3 Dec 02, 2022
trackbranch is a tool for developers that can be used to store collections of branches in the form of profiles.

trackbranch trackbranch is a tool for developers that can be used to store collections of branches in the form of profiles. This can be useful for sit

Kevin Morris 1 Oct 21, 2021
A demo titiler for Sentinel 2 Digital Twin dataset

This is a DEMO custom api built on top of TiTiler to create Web Map Tiles from the Digital Twin Sentinel-2 COG created by Sinergise

Development Seed 26 May 21, 2022
This Is A Python Program To Showcase Two Modules (Gratient And Fade)

Hellooo, It's PndaBoi Here! This Is A Python Program To Showcase Two Modules (Gratient And Fade). I Really Like Both Of These Modules So I Decided To

PndaBoi! 6 May 31, 2022
radiant discord anti nuke src leaked lol.

radiant-anti-wizz-leaked radiant discord anti nuke src leaked lol, the whole anti sucks but idc. sucks to suck thats tuff bro LMAOOOOOO join my server

ok 15 Aug 06, 2022
The Simple Google Colab Notebook to Download Files from Direct Link to Google Drive with custom name and bulk link support.

Direct Link to Google Drive (Advanced! 🔥 ) The Most Advanced yet Simple Google Colab Notebook to Download Files from Direct Link to Google Drive. 🆕

Dr.Caduceus 14 Jul 26, 2022
With Google Drive API. My computer and my phone are in love now.

Channel trought Google Drive Google Drive API In this case, "Google Drive App" is the program. To install everything you need(has some extra things),

Luis Quiñones Requelme 1 Dec 15, 2021
Telegram bot to provide Telegram user/group/channel information

Whois-TeLeTiPs Telegram bot to provide Telegram user/group/channel information Deployment Methods Heroku Config Vars API_ID : Telegram API_ID, get it

11 Oct 21, 2022
A free, minimal, lightweight, cross-platform, easily expandable Twitch IRC/API bot.

parky's twitch bot A free, minimal, lightweight, cross-platform, easily expandable Twitch IRC/API bot. Features 🔌 Connect to Twitch IRC chat! 🔌 Conn

Andreas Schneider 10 Dec 30, 2022
wyscoutapi is an extremely basic API client for the Wyscout API (v2 & v3) for Python

wyscoutapi wyscoutapi is an extremely basic API client for the Wyscout API (v2 & v3). Usage Install with pip install wyscoutapi. To connect to the Wys

Ben Torvaney 11 Nov 22, 2022
Posts word definitions on Twitter daily

Word Of The Day bot Post daily word definitions on social media. Twitter account: https://twitter.com/WordOfTheDay_B Introduction The goal of this pro

Lucas Rijllart 1 Jan 08, 2022