This is a repository for the Duke University Cloud Computing course project on Serveless Data Engineering Pipeline. For this project, I recreated the below pipeline.

Overview

AWS Data Engineering Pipeline

This is a repository for the Duke University Cloud Computing course project on Serverless Data Engineering Pipeline. For this project, I recreated the below pipeline in iCloud9 (reference: https://github.com/noahgift/awslambda):

drawing

Below are the steps of how to build this pipeline in AWS:

1๏ธโƒฃ Create a new iCloud9 environment dedicated to this project.

๐Ÿค” Need a refresher? Please check this repo.

โš ๏ธ Make sure to use name as your unique id for your items in the fang table.

2๏ธโƒฃ Create a fang table in DynamoDB and SQS queue.

You can check how to do it here.

3๏ธโƒฃ Build producer Lambda Function

  1. In iCloud9, initialize a serverless application with SAM template:

    sam init 

Inputs: 1, 2, 4, "producer"

  1. Set virtual environment and source it:

    # I called my virtual environment "comprehendProducer"
    python3 -m venv ~/.comprehendProducer
    source ~/.comprehendProducer/bin/activate
  2. Add the code for your application to app.py

  3. Add relevant packages used in your app to requirements.txt file

  4. Install requirements

     cd hello_world/
     pip install -r requirements.txt 
     cd .. 
  5. Create a repository (producer) in Elastic Container Registry (ECR) and copy its URI

  6. Build and deploy your serverless application:

    sam build 
    sam deploy --guided

    When prompted to input URI, paste the URI for the producer repository that you've just created.

  7. Create IAM Role granting Administrator Access to the Producer Lambda function.

    ๐Ÿค” Not sure how to create IAM Role? Check out this video (17 min ).

  8. Add the execution role that you created to the Producer Lambda function.

    In case you forgot how to do it:

    In AWS console: Lambda โžก๏ธ click on producer function โžก๏ธ configuration โžก๏ธ permissions โžก๏ธ Edit โžก๏ธ Select the role under Existing role.

  9. You are all set with the producer function! Now deactivate virtual environment:

    deactivate 
    cd .. 
    

4๏ธโƒฃ Create an S3 bucket and note its name

5๏ธโƒฃ Build consumer Lambda Function

Repeat steps in 3๏ธโƒฃ .

โš ๏ธ In #3 when you add the code for a consumer app to app.py, make sure to replace bucket="fangsentiment" with the name of your S3 bucket.

6๏ธโƒฃ Add triggers to Lambda Functions

๐Ÿค” Not sure how to do it? Check out this video (start times are noted below):

Producer Lambda Function: CloudWatchEvent(30 min)

Consumer Lambda Function: SQS (42 min)

7๏ธโƒฃ If all goes well, you will see sentiment results in your S3 bucket:

s3

๐Ÿ’ก Tip: If you've already deployed your Lambda function but need to edit your application, you can make the necessary edits to your app and build and deploy the app again:

sam build && sam deploy 

๐Ÿ’ก Tip: If you don't have space left on disk, you may want to remove a few docker containers that you don't use.

#list containers 
docker image ls 
# remove a container 
docker image rm <containerId>
Translator based on Google API

Yakusu Toshiko Translator based on Google API. Instance of this bot is running as @yakusubot. Features Add a plus to a language's name to show an orig

Arisu W. 2 Sep 21, 2022
Cdk-python-crud-app - CDK Python CRUD App

Welcome to your CDK Python project! You should explore the contents of this proj

Shapon Sheikh 1 Jan 12, 2022
Projeto com o objetivo de aprender o funcionamento de Consumo de APIs.

Consumindo API SuperHero Projeto com o objetivo de aprender o funcionamento de Consumo de APIs.

Deivisson Henrique 1 Dec 30, 2021
Want to play What Would Rather on your Server? Invite the bot now! ๐Ÿ˜

What is this Bot? ๐Ÿ‘€ What You Would Rather? is a Guessing game where you guess one thing. Long Description short Take this example: You typed r!rather

FSP Gang s' YT 3 Oct 18, 2021
ARKHAM X GOD MULTISPAM BOT

ARKHAM-X-GOD-MULTISPAM-BOT ๐——๐—˜๐—ฃ๐—Ÿ๐—ข๐—ฌ ๐—จ๐—ฃ๐—ง๐—ข 30 ๐—•๐—ข๐—ง๐—ฆ ๐—œ๐—ก ๐—” ๐—ฆ๐—œ๐—ก๐—š๐—Ÿ?

ArkhamXGod 2 Jan 08, 2022
A simple fun discord bot using discord.py that can post memes

A simple fun discord bot using discord.py * * Commands $commands - to see all commands $meme - for a random meme from the internet $cry - to make the

Dice Flip 2 Dec 20, 2021
Code for paper "Adversarial score matching and improved sampling for image generation"

Adversarial score matching and improved sampling for image generation This repo contains the official implementation for the ICLR 2021 paper Adversari

Alexia Jolicoeur-Martineau 114 Dec 11, 2022
Watches your earnings on EarnApp and notifies you when you earned balance or received an payout.

EarnApp-Earning-Monitor Watches your earnings on EarnApp and notifies you when you earned balance or received an payout. Installation Install Python3

Yariya 21 Oct 17, 2022
Python wrapper for Stanford CoreNLP.

stanfordcorenlp stanfordcorenlp is a Python wrapper for Stanford CoreNLP. It provides a simple API for text processing tasks such as Tokenization, Par

884 Dec 25, 2022
An unoffcial python API client for primeuploads.com

primeuploads-py An unoffcial python API wrapper for primeuploads.com Installation pip3 install primeuploads-py Usage example from prime import PrimeUp

W4RR10R 41 Dec 05, 2022
Beyonic API Python official client library simplified examples using Flask, Django and Fast API.

Beyonic API Python official client library simplified examples using Flask, Django and Fast API.

A library that allows you to easily mock out tests based on AWS infrastructure.

Moto - Mock AWS Services Install $ pip install moto[ec2,s3,all] In a nutshell Moto is a library that allows your tests to easily mock out AWS Services

Steve Pulec 6.5k Jan 02, 2023
Crypto trading bot that detects surges in the bitcoin price and executes trades.

The bot will be trading Bitcoin automatically if the price has increased by more than 3% in the last 10 minutes. We will have a stop loss of 5% and t

164 Oct 20, 2022
LavaAPI - A simple library for accepting payments and using the LAVA Wallet

This library was created to simplify the LAVA api provided on the official websi

Vlad Baccara 8 Dec 18, 2022
This is a script to export logs from AWS CloudTrail to a local file.

cloudtrail-export-logs This is a script to export logs from AWS CloudTrail to a local file. Getting Started Prerequisites python 3 boto3 pip Installin

Claick Assunรงรฃo de Oliveira 2 Jan 02, 2022
An asyncio Python wrapper around the Discord API, forked off of Rapptz's Discord.py.

Novus A modern, easy to use, feature-rich, and async ready API wrapper for Discord written in Python. A full fork of Rapptz's Discord.py library, with

Voxel Fox 60 Jan 03, 2023
Collection of script to manage WLED devices

Collection of script to manage WLED devices

Daniel Poelzleithner 4 Sep 26, 2022
A wrapper to stream information from Twitter's Full-Archive Search Endpoint

A wrapper to stream information from Twitter's Full-Archive Search Endpoint. To exploit this library, one must have approved academic research access.

Daniela Pinto Veizaga 9 Nov 28, 2022
Repo-cloner - Script takes user public liked repos and clone it to a local folder

Liked repos cloner Script takes user public liked repos and clone it to a local

Aleksei 2 Jun 18, 2022
One of Best renamer bot with python

๐ŸŒ€ One of Best renamer bot repo Please Give a โ˜† if You like This Open Source and Don't Forget to Follow Me On Github For More Repos And Codes. Scrappe

1 Dec 14, 2021