Jupyter notebooks and AWS CloudFormation template to show how Hudi, Iceberg, and Delta Lake work

Overview

Modern Data Lake Storage Layers

This repository contains supporting assets for my research in modern Data Lake storage layers like Apache Hudi, Apache Iceberg, and Delta Lake.

Specifically, there's a CloudFormation template to create an EMR cluster and EMR Studio with the necessary requirements and Jupyter notebooks with the example walkthroughs.

You can view the corresponding blog post and video

Pre-requisites

You'll need an AWS Account in which you have administrator privileges and the ability to deploy a CloudFormation template. The template will create an EMR Cluster and S3 bucket that will incur charges - be sure to either shut down the cluster when done or delete the CloudFormation stack. In order to delete the CloudFormation stack, you'll need to:

  • Manually delete any EMR Studio Workspaces you created
  • Manually empty the S3 bucket created by CloudFormation
  • Manually delete the VPC created by CloudFormation due to auto-created rules

Overview

The included CloudFormation template creates a new VPC and EMR Cluster for you to be able to run the notebooks. An EMR Studio is also created and you can find the Studio URL in the Outputs tab of your CloudFormation Stack.

Once the stack is done creating, you'll need to navigate to EMR Studio and create a new workspace attached to the "data-lakes" cluster.

Inside the workspace you either upload each notebook individually from the notebooks/ folder or simply connect to this repository by using the "Git" icon on the left-hand side.

Me and @nathanmargni did a small analysis on what are the best strategies to win more games of League of Legends.

Me and @nathanmargni did a small analysis on what are the best strategies to win more games of League of Legends.

Christian Berchtold 2 Jan 19, 2022
YuuScource - A Discord bot made with Pycord

Yuu A Discord bot made with Pycord Features Not much lol • Easy to use commands

kekda 9 Feb 17, 2022
This is a Innexia Group Manager Bot with many features

⚡ Innexia ⚡ A Powerful, Smart And Simple Group Manager ... Written with AioGram , Pyrogram and Telethon... Available on Telegram as @Innexia ❤️ Suppor

TeamDeeCode 84 Jun 04, 2022
Discord bot to administer IITD Study Servers (unofficial)

IITD-Bot Discord bot to administer IITD'20 Acad Server Commands hello to check if bot is online ?help to display this message ?set kerberos to set y

Aditya Singh 47 Dec 19, 2022
OAN Music - Highly advanced User Music Bot

ཧᜰ꙰ꦿ➢𝐎𝐀𝐍༒☛ 🎧 Advanced 𝐎𝐀𝐍 Music bot. 🔗 𝐏𝐨𝐰𝐞𝐫𝐞𝐝 𝐛𝐲 : ➢𝐀ttitude

Attitude king 5 Feb 25, 2022
Bitcoin tracker hecho con python.

Bitcoin Tracker Precio del Bitcoin en tiempo real. Script simple hecho con python. Rollercoin RollerCoin es un juego en el que puedes ganar bitcoin (y

biyivi 3 Jan 04, 2022
An Advanced Telegram Bot to Play Radio & Music in Voice Chat. This is Also The Source Code of The Bot Which is Being Used For Playing Radio in @AsmSafone Channel ❤️

Telegram Radio Player V3 An Advanced Telegram Bot to Play Nonstop Radio/Music/YouTube Live in Channel or Group Voice Chats. This is also the source co

SAF ONE 421 Jan 05, 2023
High-Resolution Differential Z-Belt Mod for V0 (with optional Kirigami support)

V0-DBM This is a high-resolution differential pulley system belt mod for the Z-axis on Voron 0 with optional Kirigami Bed support. NOTE: Alpha version

Simon Küppers 11 Jan 07, 2023
OpenEmu Discord Rich Presence provided with Python!

A simple application that provides your current OpenEmu game as an RPC state in Discord via PyPresence. How to use Unzip and open the latest x86_64 ve

Deltaion Lee 6 May 30, 2022
The first open-source PyTgCalls-based project.

Calls Music — Telegram bot + userbot for streaming audio in group calls ✍🏻 Requirements FFmpeg Python 3.7+ 🚀 Deployment 🛠 Configuration Copy exampl

Calls Music 74 Nov 19, 2022
Telegram bot to stream videos in telegram voicechat for both groups and channels

Telegram bot to stream videos in telegram voicechat for both groups and channels. Supports live streams, YouTube videos and telegram media. With record stream support, Schedule streams, and many more

ALBY 9 Feb 20, 2022
Telegram üzerinden paylaşılan kısa linkleri geçmenin daha hızlı bir yolu

Telegram Url skipper Telegramda paylaşılan kısa linkleri geçmenin daha hızlı bir yolu · Hata Raporla · Öneri Yap İçerik Tablosu Kurulum Kullanım Lisan

WarForPeace 6 Oct 07, 2022
A simple way to create a request to the coinpayment API with a valid HMAC using your private key and command

Coinpayments Verify TXID Created for Astral Discord bot A simple way to create a request to the coinpayment API with a valid HMAC using your private k

HellSec 1 Nov 07, 2022
S3-cleaner - A Python script attempts to delete the all objects/delete markers/versions from specific S3 bucket

Remove All Objects From S3 Bucket This Python script attempts to delete the all

9 Jan 27, 2022
Pydf: A modular Telegram Bot which provides Pdf Tools using PyPdf2

pyDF-Bot 🌍 Pydf - Pyrogram Document File Bot, a modular Telegram Bot which prov

HyDrix 2 Feb 18, 2022
TgMusicBot is a telegram userbot for playing songs in telegram voice calls based on Pyrogram and PyTgCalls.

TgMusicBot [Stable] TgMusicBot is a telegram userbot for playing songs in telegram voice calls based on Pyrogram and PyTgCalls. Commands !start / !hel

Kürşad 21 Dec 25, 2022
A discord bot to assist you when playing phasmophobia.

phasbot A discord bot to assist you when playing phasmophobia. Add phasbot to your server here! Bot Commands ?help - shows commands ?info [ghost name]

1 Dec 22, 2021
Cogs version of iso6.9 with the help of thatOneArchUser

iso6.9-cogs (debloated) This is a cogs version of iso6.9 by αrchιshα#5518. iso6.9 is a Discord bot written in Python and is used to make your Discord

Kamilla Youver 2 Jun 10, 2022
An async python wrapper to interact with the Steam API and its CMs

steam.py A modern, easy to use, and async ready package to interact with the Steam API. Heavily inspired by discord.py and borrowing functionality fro

James Hilton-Balfe 90 Dec 15, 2022
Instagram bot for promoting ROKA trainee soldier(just like me)'s consolation letters.

Instagram_bot (필자를 포함한) 모든 대한민국 훈련병들을 위한 인스타그램 인편지기입니다. Instagram bot for promoting ROKA trainee soldier(just like me)'s consolation letters. 들어가기 (Ge

Lee, Jongjun 2 Nov 21, 2021