This is a repository for the Duke University Cloud Computing course project on Serveless Data Engineering Pipeline. For this project, I recreated the below pipeline.

Overview

AWS Data Engineering Pipeline

This is a repository for the Duke University Cloud Computing course project on Serverless Data Engineering Pipeline. For this project, I recreated the below pipeline in iCloud9 (reference: https://github.com/noahgift/awslambda):

drawing

Below are the steps of how to build this pipeline in AWS:

1️⃣ Create a new iCloud9 environment dedicated to this project.

🤔 Need a refresher? Please check this repo.

⚠️ Make sure to use name as your unique id for your items in the fang table.

2️⃣ Create a fang table in DynamoDB and SQS queue.

You can check how to do it here.

3️⃣ Build producer Lambda Function

  1. In iCloud9, initialize a serverless application with SAM template:

    sam init 

Inputs: 1, 2, 4, "producer"

  1. Set virtual environment and source it:

    # I called my virtual environment "comprehendProducer"
    python3 -m venv ~/.comprehendProducer
    source ~/.comprehendProducer/bin/activate
  2. Add the code for your application to app.py

  3. Add relevant packages used in your app to requirements.txt file

  4. Install requirements

     cd hello_world/
     pip install -r requirements.txt 
     cd .. 
  5. Create a repository (producer) in Elastic Container Registry (ECR) and copy its URI

  6. Build and deploy your serverless application:

    sam build 
    sam deploy --guided

    When prompted to input URI, paste the URI for the producer repository that you've just created.

  7. Create IAM Role granting Administrator Access to the Producer Lambda function.

    🤔 Not sure how to create IAM Role? Check out this video (17 min ).

  8. Add the execution role that you created to the Producer Lambda function.

    In case you forgot how to do it:

    In AWS console: Lambda ➡️ click on producer function ➡️ configuration ➡️ permissions ➡️ Edit ➡️ Select the role under Existing role.

  9. You are all set with the producer function! Now deactivate virtual environment:

    deactivate 
    cd .. 
    

4️⃣ Create an S3 bucket and note its name

5️⃣ Build consumer Lambda Function

Repeat steps in 3️⃣ .

⚠️ In #3 when you add the code for a consumer app to app.py, make sure to replace bucket="fangsentiment" with the name of your S3 bucket.

6️⃣ Add triggers to Lambda Functions

🤔 Not sure how to do it? Check out this video (start times are noted below):

Producer Lambda Function: CloudWatchEvent(30 min)

Consumer Lambda Function: SQS (42 min)

7️⃣ If all goes well, you will see sentiment results in your S3 bucket:

s3

💡 Tip: If you've already deployed your Lambda function but need to edit your application, you can make the necessary edits to your app and build and deploy the app again:

sam build && sam deploy 

💡 Tip: If you don't have space left on disk, you may want to remove a few docker containers that you don't use.

#list containers 
docker image ls 
# remove a container 
docker image rm <containerId>
Irenedao-nft-generator - Original scripts used to generate IreneDAO NFTs

IreneDAO NFT Generator Scripts to generate IreneDAO NFT. Make sure you have Pill

libevm 60 Oct 27, 2022
An Advanced Telegram Bot to Play Radio & Music in Voice Chat. This is Also The Source Code of The Bot Which is Being Used For Playing Radio in @AsmSafone Channel ❤️

Telegram Radio Player V3 An Advanced Telegram Bot to Play Nonstop Radio/Music/YouTube Live in Channel or Group Voice Chats. This is also the source co

SAF ONE 421 Jan 05, 2023
An API wrapper for Henrik's Unofficial VALORANT API

ValorantAPI.py An API wrapper for Henrik's Unofficial VALORANT API Warning!! This project is still in beta and only contains barely anything yet. If y

Jakkaphat Chalermphanaphan 0 Feb 04, 2022
An api, written in Python, for Investopedia's paper trading stock simulator.

investopedia-trading-api An API, written in Python, for Investopedia's paper trading stock simulator. Pull requests welcome. This library is now Pytho

Kirk Thaker 178 Jan 06, 2023
With this simple app you can customize your presence in Discord.

Discord Rich Presence This a simple console app which can customize your Discord Presence easily, with only one config file! Setup Go to Discord Devel

Mai 1 Jan 06, 2022
Python bindings for LibreTranslate

Python bindings for LibreTranslate

Argos Open Tech 42 Jan 03, 2023
Un petit tool qui est la pour envoier des message avec des webhook en bêta

📎 Webhook-discord Le but de se tool c'est que tu peux envoier vos webhook discord sur vos serveur et les customiser Pour lancer le projet il faut avo

2 Oct 10, 2021
Want to play What Would Rather on your Server? Invite the bot now! 😏

What is this Bot? 👀 What You Would Rather? is a Guessing game where you guess one thing. Long Description short Take this example: You typed r!rather

FSP Gang s' YT 3 Oct 18, 2021
Discord Token Nuker With Python

Discord token nuker a.k.a A$$Fvcker Setup For installing the requirements do this: pip install -r requirements.txt To start the Token nuker run this

PR3C14D0 8 Sep 22, 2022
Robot Swerve Test Public With Python

Robot-Swerve-Test-Public The codebase for our swerve drivetrain prototype robot.

1 Jan 09, 2022
A bot written in Python to automate attending classes on MyClass (Codetantra).

codetantrabot This is python program to attend class on myclass(codetantra) Prerequisites You should have Python3 and Pip installed on your system Run

Aniket Kumar 1 Feb 08, 2022
Instagram Bot posting earthquakes with magnitude greater than or equal to 3.5.

Instagram Bot posting earthquakes with magnitude greater than or equal to 3.5

Alican Yüksel 4 Aug 22, 2022
Async client API for the Telegram Group Calls

PyTgCalls This project allow to make Telegram group call with MTProto Api using Pyrogram and WebRTC, this is possible thanks to the power of NodeJS's

185 Jan 03, 2023
Chatbot with python code!

Chatbot Python Chatbot with python! How to Run Installation requirements. pip install -r requirements.txt Sample Chatbot The required files must be d

Mohammad Dori 3 Jul 15, 2022
python based bot Sends notification to your telegram whenever a new video is released on a youtube channel!

YTnotifier python based bot Sends notification to your telegram whenever a new video is released on a youtube channel! REQUIREMENTS telethon python-de

Mohamed Rizad 6 Jul 23, 2022
TeslaPy - A Python implementation based on unofficial documentation of the client side interface to the Tesla Motors Owner API

TeslaPy - A Python implementation based on unofficial documentation of the client side interface to the Tesla Motors Owner API, which provides functiona

Tim Dorssers 233 Dec 30, 2022
Frwdit-V1 - A Simple Bot can copy any media to a private channel provided

📠 Auto Forward V2 A Simple Bot can copy any media to a private channel provided

FUCK OFF ANE 3 Dec 03, 2022
DEPRECATED - Official Python Client for the Discogs API

⚠️ DEPRECATED This repository is no longer maintained. You can still use a REST client like Requests or other third-party Python library to access the

Discogs 483 Dec 31, 2022
Unofficial Meteor Client wiki

Welcome to the Unofficial Meteor Client wiki! Meteor FAQs | A rewritten and better FAQ page. Installation Guide | A guide on how to install Meteor Cli

Anti Cope 0 Feb 21, 2022
A wrapper for The Movie Database API v3 and v4 that only uses the read access token (not api key).

fulltmdb A wrapper for The Movie Database API v3 and v4 that only uses the read access token (not api key). Installation Use the package manager pip t

Jacob Hale 2 Sep 26, 2021