Pyspark sam - Analyze Big Sequence Alignments with PySpark in AWS EMR

Related tags

Third-party APIs Wrapperspyspark_sam
Overview

pyspark_sam

This repo hosts my code for the article "Analyze Big Sequence Alignments with PySpark in AWS EMR".

Prerequisite

  1. Spark

  2. AWS CLI

  3. AWS Account

Run

Follow the instruction in the article. Once you have uploaded the files into your S3 bucket, run

aws emr create-cluster --name "Spark_step_pip" \
    --release-label emr-6.5.0 \
    --applications Name=Spark \
    --log-uri s3://[your_S3_bucket]/logs/ \
    --instance-type m5.xlarge \
    --instance-count 3 \
    --bootstrap-actions Path=s3://[your_S3_bucket]/emr_bootstrap.sh \
    --use-default-roles --auto-terminate \
    --steps "Type=Spark,Name=SparkProgram,ActionOnFailure=CONTINUE,Args=[--deploy-mode,cluster,--master,yarn,--py-files,s3://[your_S3_bucket]/helper_function.py,s3://[your_S3_bucket]/spark_3mer.py,s3://[your_S3_bucket]/test.sam,[your_S3_bucket],sankey.json]"

When the job finishes, download the sankey.json. And run this command to visualize:

python sankey.py sankey.json

Authors

  • Sixing Huang - Concept and Coding

License

This project is licensed under the MIT License - see the LICENSE file for details

Owner
Sixing Huang
A triple Neo4j certified data scientist. I am currently working at BGI in Shenzhen.
Sixing Huang
GitHub Repository
Python Capfire API wrapper

General CampfireAPI based on Campfire web. Install pip install Campfire-API Quickstart Use it without login: from campfire_api import CampfireAPI cf

Ghost 0 Jan 03, 2022
Telegram tools

Telegram-Tools Telegram tools. Explanation English | 中文 Features Export group memebrs Add users to the group Send message to users Setup API Open http

4 Apr 02, 2022
A simple waybar module to display the status of the ICE you are currently in using the ICE Portals JSON API.

waybar-iceportal A simple waybar module to display the status of the ICE you are currently in using the ICE Portals JSON API. Installation Ensure pyth

Moritz 7 Aug 26, 2022
Telegram bot to provide Telegram user/group/channel information

Whois-TeLeTiPs Telegram bot to provide Telegram user/group/channel information Deployment Methods Heroku Config Vars API_ID : Telegram API_ID, get it

11 Oct 21, 2022
Hello i am TELEGRAM GROUP MANAGEMENT BOT MY NAME IS Evil-Inside ⚡ i have both amazing modules

Evil-Inside DEMO BOT - Evil-Inside Hello i am TELEGRAM GROUP MANAGEMENT BOT MY NAME IS Evil-Inside ⚡ i have both amazing modules ℂ𝕆ℕ𝕋𝔸ℂ𝕋 𝕄𝔼 𝕆ℕ

PANDITHAN 52 Nov 20, 2022
A plugin for modmail-bot for stealing,making ,etc emojis

EmojiPlugin for the Modmail-bot My first plugin .. its very Basic I will make more and better too Only 3 commands for now emojiadd-supports .jpg, .png

1 Dec 28, 2021
Crypto trading bot that detects surges in the bitcoin price and executes trades.

The bot will be trading Bitcoin automatically if the price has increased by more than 3% in the last 10 minutes. We will have a stop loss of 5% and t

164 Oct 20, 2022
The first open-source PyTgCalls-based project.

Calls Music — Telegram bot + userbot for streaming audio in group calls ✍🏻 Requirements FFmpeg Python 3.7+ 🚀 Deployment 🛠 Configuration Copy exampl

Calls Music 74 Nov 19, 2022
WebCash is an experimental new electronic cash ("e-cash") that enables decentralized and instant payments to anyone

Webcash WebCash is an experimental new electronic cash ("e-cash") that enables decentralized and instant payments to anyone, anywhere in the world. Us

Bryan Bishop 24 Dec 11, 2022
A Python wrapper for discord slash-commands, designed to extend discord.py.

dislash.py An extending library for discord.py that allows to build awesome slash-commands. ⭐

173 Dec 19, 2022
Data portal client and server for NMDC.

NMDC Server and Client Portal Getting started with Docker install ldc install submodules via git submodule update --init --recursive In order to popul

National Microbiome Data Collaborative 7 Dec 14, 2022
GroupMenter : New Telegram Group Manager Bot🔸Fast 🔸Python🔸Pyrogram 🔸

GroupMenter An PowerFull Group Manager Bot. Written In Pytelethon. Info • A modular Telegram Python bot running on python3. • Can be found on telegram

Group Menter 24 Jun 28, 2022
I was sick of having to hand my friends my phone, so I gave my Spotify some SMS features!

SMSpotifY Just a little tool so that my friends can text a phone number and add to my spotify queue for parties and such:) Features Roles / Access Con

Sara 2 Jan 17, 2022
Written in Python, freezed into stand-alone executable with PyInstaller. This app will make sure you stay in New World without getting kicked for inactivity.

New World - AFK Written in Python, freezed into stand-alone executable with PyInstaller. This app will make sure you stay in New World without getting

Rodney 5 Oct 31, 2021
A Twitter Bot that retweets and likes tweets with the hashtag #girlscriptwoc and #girlscript, and also follows the user.

GirlScript Winter of Contributing Twitter Bot A Twitter Bot that retweets and likes tweets with the hashtag #girlscriptwoc and #girlscript, and also f

Pranay Gupta 9 Dec 15, 2022
A very basic starter bot based on CryptoKKing with a small balance

starterbot A very basic starter bot based on CryptoKKing with a small balance, use at your own risk. I have since upgraded this script significantly a

Danny Kendrick 2 Dec 05, 2021
Pydf: A modular Telegram Bot which provides Pdf Tools using PyPdf2

pyDF-Bot 🌍 Pydf - Pyrogram Document File Bot, a modular Telegram Bot which prov

HyDrix 2 Feb 18, 2022
A modular Telegram group management bot running with Python based on Pyrogram.

A modular Telegram group management bot running with Python based on Pyrogram.

Jefanya Efandchris 1 Nov 14, 2022
The best Discord bot, created for r/Jailbreak

Bloo Setup instructions These instructions assume you are on macOS or Linux. Windows users, good luck. With Docker (recommended!) You will need the fo

GIR 33 Dec 16, 2022
Non official, but friendly QvaPay library for the Python language.

Python SDK for the QvaPay API Non official, but friendly QvaPay library for the Python language. Setup You can install this package by using the pip t

Carlos Lugones 17 Nov 25, 2022