Pyspark sam - Analyze Big Sequence Alignments with PySpark in AWS EMR

Overview

pyspark_sam

This repo hosts my code for the article "Analyze Big Sequence Alignments with PySpark in AWS EMR".

Prerequisite

  1. Spark

  2. AWS CLI

  3. AWS Account

Run

Follow the instruction in the article. Once you have uploaded the files into your S3 bucket, run

aws emr create-cluster --name "Spark_step_pip" \
    --release-label emr-6.5.0 \
    --applications Name=Spark \
    --log-uri s3://[your_S3_bucket]/logs/ \
    --instance-type m5.xlarge \
    --instance-count 3 \
    --bootstrap-actions Path=s3://[your_S3_bucket]/emr_bootstrap.sh \
    --use-default-roles --auto-terminate \
    --steps "Type=Spark,Name=SparkProgram,ActionOnFailure=CONTINUE,Args=[--deploy-mode,cluster,--master,yarn,--py-files,s3://[your_S3_bucket]/helper_function.py,s3://[your_S3_bucket]/spark_3mer.py,s3://[your_S3_bucket]/test.sam,[your_S3_bucket],sankey.json]" 

When the job finishes, download the sankey.json. And run this command to visualize:

python sankey.py sankey.json

Authors

  • Sixing Huang - Concept and Coding

License

This project is licensed under the MIT License - see the LICENSE file for details

Owner
Sixing Huang
A triple Neo4j certified data scientist. I am currently working at BGI in Shenzhen.
Sixing Huang
One of Best renamer bot with python

🌀 One of Best renamer bot repo Please Give a ☆ if You like This Open Source and Don't Forget to Follow Me On Github For More Repos And Codes. Scrappe

1 Dec 14, 2021
Asynchronous Python Wrapper for the GoFile API

Asynchronous Python Wrapper for the GoFile API

Gautam Kumar 22 Aug 04, 2022
Estimate the total emissions for popular CryptoArt platforms.

cryptoart-footprint Estimate the total CO2 footprint for popular CryptoArt platforms. The goal is to accurately quantify the ecological damage of Ethe

Kyle McDonald 182 Oct 12, 2022
An Open-Source Discord bot created to provide basic functionality which should be in every discord guild. We use this same bot with additional configurations for our guilds.

A Discord bot completely written to be taken from the source and built according to your own custom needs. This bot supports some core features and is

Tesseract Coding 14 Jan 11, 2022
🎀 First and most powerfull open source clicktune botter

CTB 🖤 Follow me here: Discord | YouTube | Twitter | Github 🐺 Features: /* *- The first *- Fast *- Proxy support: http/s, socks4/5, premieum (w

Iтѕ_Ѵιcнч#1337 22 Aug 29, 2022
A very basic starter bot based on CryptoKKing with a small balance

starterbot A very basic starter bot based on CryptoKKing with a small balance, use at your own risk. I have since upgraded this script significantly a

Danny Kendrick 2 Dec 05, 2021
ByDiego Token Grabber is a Discord Stealer

ByDiego Token Grabber is a Discord Stealer. This way you can get too much information from x person if you pass it on and open it

zByDiegoM.T 4 Mar 11, 2022
a list of disposable and temporary email address domains

List of disposable email domains This repo contains a list of disposable and temporary email address domains often used to register dummy users in ord

1.6k Jan 08, 2023
Aria/qBittorrent Telegram mirror/leech bot

This is a Telegram Bot written in Python for mirroring files on the Internet to your Google Drive or Telegram. Based on python-aria-mirror-bot Feature

28 Dec 25, 2022
Roaster - this gui app + program bundle roasts.

Roaster - this gui app + program bundle roasts.

Harsh ADV) 1 Jan 04, 2022
Telegram File Renamer Bot

RENAMER_BOT Telegram File Renamer Bot Configs TG_BOT_TOKEN - Get bot token from @BotFather API_ID - From my.telegram.org API_HASH - From my.telegram.o

Lntechnical 37 Dec 27, 2022
DonLee Robot

🤖 𝐃𝐎𝐍 𝐋𝐄𝐄 𝐑𝐎𝐁𝐎𝐓 𝐕𝟐 🤖 👋 Hey Muhammed, Iam DonLee RoBoT Make me an admin for your group and channel then connect me.... 🎉 🙂 To build a

Muhammed 27 Dec 01, 2022
A Telegram Repo For Devs To Controll The Bots Under Maintenance.This Bot Is For Developers, If Your Bot Is Down, Use This Repo To Give Your Dear Subscribers Some Support By Providing Them Response.

Maintenance Bot A Telegram Repo For Devs To Controll The Bots Under Maintenance About This Bot This Bot Is For Developers, If Your Bot Is Down, Use Th

Vɪᴠᴇᴋ 47 Dec 29, 2022
Discord Account Generator that will create Account with hCaptcha bypass. Using socks4 proxies

Account-Generator [!] This was made for education. Please use socks4 proxies for nice experiences. [!] Please install these modules - "pip3 install ht

RyanzSantos 10 Feb 23, 2022
Random-backlog-tweet - Pick a page from a sitemap at random and prep a tweet button for it

Random-backlog-tweet - Pick a page from a sitemap at random and prep a tweet button for it

Paul O'Leary McCann 0 Dec 01, 2022
Example-bot-discord - Example bot discord xD

example-python-bot-discord Clone this repository Grab a token on Discord's devel

Amitminer 1 Mar 14, 2022
A replacement for Reddit /r/copypasta CummyBot2000 with extra measures to avoid it being banned.

CummyBot1984 A replacement for Reddit /r/copypasta's CummyBot2000 with extra measures to respect Reddit's API rules. Features Copies and replies to ev

2 Feb 21, 2022
One destination for all the developer's learning resources.

DevResources One destination for all the developer's learning resources. Find all of your learning resources under one roof and add your own. Live ✨ Y

Gaurav Sharma 33 Oct 21, 2022
LHXP・Official "LH - Cyber Security" Discord Leveling-Bot

LHXP・Official "LH - Cyber Security" Discord Leveling-Bot Based on nsde/NOVΛLIX Feature Overview /clear @user Requires admin permission Purges all XP

Felix・onlix 2 Mar 09, 2022
Discord Voice Channel Automatic Online

Discord-Selfbot-voice Features: Discord Voice Channel Automatic Online FAQ Q: How can I obtain my token? A: 1. How to obtain your token in android 2.

Pranav Ajay 3 Oct 31, 2022