Pyspark sam - Analyze Big Sequence Alignments with PySpark in AWS EMR

Related tags

Third-party APIs Wrapperspyspark_sam
Overview

pyspark_sam

This repo hosts my code for the article "Analyze Big Sequence Alignments with PySpark in AWS EMR".

Prerequisite

  1. Spark

  2. AWS CLI

  3. AWS Account

Run

Follow the instruction in the article. Once you have uploaded the files into your S3 bucket, run

aws emr create-cluster --name "Spark_step_pip" \
    --release-label emr-6.5.0 \
    --applications Name=Spark \
    --log-uri s3://[your_S3_bucket]/logs/ \
    --instance-type m5.xlarge \
    --instance-count 3 \
    --bootstrap-actions Path=s3://[your_S3_bucket]/emr_bootstrap.sh \
    --use-default-roles --auto-terminate \
    --steps "Type=Spark,Name=SparkProgram,ActionOnFailure=CONTINUE,Args=[--deploy-mode,cluster,--master,yarn,--py-files,s3://[your_S3_bucket]/helper_function.py,s3://[your_S3_bucket]/spark_3mer.py,s3://[your_S3_bucket]/test.sam,[your_S3_bucket],sankey.json]"

When the job finishes, download the sankey.json. And run this command to visualize:

python sankey.py sankey.json

Authors

  • Sixing Huang - Concept and Coding

License

This project is licensed under the MIT License - see the LICENSE file for details

Owner
Sixing Huang
A triple Neo4j certified data scientist. I am currently working at BGI in Shenzhen.
Sixing Huang
GitHub Repository
Recursive-Bucket-Sort - An efficient sorting algorithm (implemented in Python) inspired by the Bucket Sort and the Pigeonhole Sort

Recursive Bucket Sorting Algorithm An algorithm (implemented here in Python) mai

Joshua McDonagh 1 Jan 24, 2022
An API Client package to access the APIs for NBA.com

nba_api An API Client package to access the APIs for NBA.com Development Version: v1.1.9 nba_api is an API Client for www.nba.com. This package is mea

Swar Patel 1.4k Jan 01, 2023
A delivery protection and notification system

DeliveryProtect This project builds a delivery protection and notification system, based on integration of Arduino Uno and Raspberry Pi 4. The codes a

2 Dec 13, 2021
Código para trabalho com o dataset Wine em Python

Um perceptron multicamadas (MLP) é uma rede neural artificial feedforward que gera um conjunto de saídas a partir de um conjunto de entradas. Um MLP é

Hemili Beatriz 1 Jan 08, 2022
Wrapper for wttr.in weather forecast.

pywttr Wrapper for wttr.in weather forecast. Asynchronous version here. Installation pip install pywttr Example This example prints the average temper

Almaz 6 Dec 25, 2022
Calendars for various securities exchanges.

IMPORTANT NOTE This package is currently unmaintained as the sponsor, quantopian, is going through corporate changes. As such there is a fork of this

Quantopian, Inc. 545 Jan 07, 2023
Ark API Wrapper in Python

Pythark Ark API Wrapper in Python. Built with Python Requests Installation Pythark uses Arky to create a new transaction, if you want to use this feat

Jolan 14 Mar 11, 2021
Weather Tracker, made with Python using Open Weather API

Weather Tracker Weather Tracker, made with Python using Open Weather API

Sahil Kumar 1 Feb 07, 2022
Discord bot to monitor collection of mods on the Steam Workshop and notify on update to selected discord server via Nextcordbot API.

Steam-Workshop-Monitor Discord bot to monitor collection of mods on the Steam Workshop and notify on update to selected Discord channel via Nextcordbo

7 Nov 03, 2022
I was sick of having to hand my friends my phone, so I gave my Spotify some SMS features!

SMSpotifY Just a little tool so that my friends can text a phone number and add to my spotify queue for parties and such:) Features Roles / Access Con

Sara 2 Jan 17, 2022
ImaginaryTicketing is a simple ticketing system for running CTF Competitions on discord.

ImaginaryTicketing ImaginaryTicketing is a simple ticketing system for running CTF Competitions on discord. Be sure to checkout ImaginaryCTF. See docs

GudOreos 8 Jul 18, 2022
Report-snapchat - Report Snapchat acc with python

report-snapchat Report Snapchat acc Report users on Snapchat about the tool : 4

17 Dec 01, 2022
A Python bot that uses the Reddit API to send users inspiring messages.

AnonBot By Edric Antoine A Python bot that uses the Reddit API to send users inspiring messages. When a message includes 'What would Anon do?', the bo

1 Jan 05, 2022
🐍 VerificaC19 SDK implementation for Python

VerificaC19 Python SDK 🐍 VerificaC19 SDK implementation for Python. Requirements Python version = 3.7 Make sure zbar is installed in your system For

Lotrèk 10 Jan 14, 2022
A Powerful Discord Webhook spammer

RocketHook | The discord spammer Some links: Telegram | Github FEATURES 🤖 Hide your ass and be a good robot ! Full proxies support HTTP/S, SOCKS4/5 S

Or 2 Feb 27, 2022
A Discord webhook spammer made in Python.

A Python made Discord webhook spammer usually used for token loggers to spam them/delete them original by cattyn I only made it so u can change the avatar to whatever u want instead of it being hardc

notperry1234567890 15 Dec 15, 2021
A modular Telegram Python bot running on python3 with a sqlalchemy, redislab, mongo database, telethon, and pyrogram.

Zeldris Robot A modular Telegram Python bot running on python3 with a sqlalchemy, redislab, mongo database, telethon, and pyrogram. How to set up/depl

IDNCoderX 42 Dec 21, 2022
Make a command interpreter that manages AirBnb objects

AirBnB Clone Project Description This is part 1 of our AirBnb Clone project. The purpose of this project is to make a command interpreter that manages

Firdaus H. Salim 1 Nov 14, 2021
Pincer-bot-template - A template for a Discord bot created using the Pincer library

Pincer Discord Bot Template (Python) WARNING: Pincer is still in its alpha/plann

binds 2 Mar 17, 2022
A (probably) working Kik name checker

KikNameChecker !THIS ONLY CHECKS WS2.KIK.COM ENDPOINT! \ Will add user inputted endpoints thing \ A (probably) working Kik name checker Started as a s

insert edgy and cool name 1 Dec 17, 2022