A tool for batch processing large fasta files and accompanying metadata table to upload to repositories via API

Overview

Fasta Uploader

A tool for batch processing large fasta files and accompanying metadata table to repositories via API

The python fasta_uploader.py script breaks large fasta files (e.g. 500mb) and related (one-to-one) tab-delimited sample contextual data into smaller batches of 1000 or some specified # of records which can then be uploaded to a given sequence repository if an API endpoint is selected. Currently there is one option for the API interface: VirusSeq.

This tool is developed by the SFU Centre for Infectious Disease Epidemiology and One Health in conjunction with VirusSeq and it works well with DataHarmonizer!

Authors: Damion Dooley, Nithu Sara John

Details

Given a fasta file and a sample metadata file with a column that matches to fasta file record identifiers, break both into respective sets of smaller batches of records which are submitted to an API for processing.

Processing is three step:

  1. Construct batches of files. Since only two files are read and parsed in one go, processing of them is reliable after that point, so no further error reporting is needed during the batch file generation process.

    1. Importantly, if rerunning fasta_batch_submit.py, this step will be skipped unless -f --force parameter is run. Currently input files are still required in this case.
  2. IF API option is included, submit each *.queued.fasta batch to API, wait for it to finish or error out (capture error report) and proceed to next batch.

    1. Some types of error trigger sudden death, i.e. sys.exit() because they would also occur in subsequent API batch calls. For example missing tabular data column names will trigger an exit. Once resolved, rerun with -r to force regeneration of output files.
    2. There is an option to just try submitting one of the batches, e.g. the first one, via "-n 0" parameter. This allows error debugging of just the first batch. Once error patterns are determined, those that apply to remaining source contextual data can be applied, and first batch removed from source fasta and contextual data files, and the whole batching can be redone using -r reset, or by manually deleting the output files and rerunning.
  3. The processing status of existing API requests is reported from the API server end. Some may be queued by the API server, others may have been processed successfully, and others may have line-by-line errors in field content that are converted by fasta_uploader.py into new [output file batch.#].queued.fasta and [output batch.#].queued.tsv files which can be edited and then submitted back to the API by rerunning the program with the same command line parameters.

Requires Biopython and Requests modules

  • "pip install biopython"
  • "pip install requests"

Usage

Run the command in a folder with the appropriate input files, and output files can be generated there too. Rerun it in the same folder to incrementally fix any submission errors and then restart submission.

python fasta_uploader.py [options]

Options:

-h, --help
  Show this help message and exit.
-f FASTA_FILE, --fasta=FASTA_FILE
  Provide a fasta file name.
-m METADATA_FILE, --metadata=METADATA_FILE
  Provide a COMMA .csv or TAB .tsv delimited sample contextual data file name.
-b BATCH, --batch=BATCH
  Provide number of fasta records to include in each batch. Default is 1000.
-o OUTPUT_FILE, --output=OUTPUT_FILE
  Provide an output file name/path.
-k KEY_FIELD, --key=KEY_FIELD
  Provide the metadata field name to match to fasta record identifier.
-n BATCH_NUMBER, --number=BATCH_NUMBER
  Process only given batch number to API instead of all batches.

Parameters involved in optional API call:

-a API, --api=API     
  Provide the target API to send data too.  A batch submission job will be initiated for it. Default is "VirusSeq_Portal".
-u API_TOKEN, --user=API_TOKEN
  An API user token is required for API access.
-d, --dev
  Test against a development server rather than live one.  Provide an API endpoint URL.
 -s, --short
  Report up to given # of fasta record related errors for each batch submission.  Useful for taking care of repeated errors first based on first instance.
 -r, --reset
  Regenerate all batch files and begin API resubmission process even if batch files already exist under given output file pattern.

For example:

python fasta_uploader.py -f "consensus_final.fasta" -m "final set 1.csv" -k "fasta header name" -a VirusSeq_Portal -u ENTER_API_KEY_HERE

This will convert consensus_final.fasta and related final set 1.csv contextual data records into batches of 1000 records by default, and will begin submitting each batch to the VirusSeq portal.

python fasta_uploader.py -f "consensus_final.fasta" -m "final set 1.csv" -n 0 -k "fasta header name" -a VirusSeq_Portal -u ENTER_API_KEY_HERE

Like the above but only first batch is submitted so that one can see any errors, and if they apply to all batches, can fix them in original "final set 1.csv" file. Once batch 0 is fixed, all its records can be removed from the consensus_final.fasta and final set 1.csv.csv files, and the whole job can be resubmitted.

Owner
Centre for Infectious Disease and One Health
Hsiao Laboratory at Simon Fraser University
Centre for Infectious Disease and One Health
A bot discord that can create directories, file, rename, move, navigate throw directories etc....

File Manager Discord What is the purpose of this program ? This program is made for a Discord bot. Its purpose is to organize the messages sent in a c

1 Feb 02, 2022
pydicom - Read, modify and write DICOM files with python code

pydicom is a pure Python package for working with DICOM files. It lets you read, modify and write DICOM data in an easy "pythonic" way.

DICOM in Python 1.5k Jan 04, 2023
Simple addon to create folder structures in blender.

BlenderCreateFolderStructure Simple Add-on to create a folder structure in Blender. Installation Download BlenderCreateFolderStructure.py Open Blender

Dominik Strasser 2 Feb 21, 2022
A tiny Python library for writing multi-channel TIFF stacks.

xtiff A tiny Python library for writing multi-channel TIFF stacks. The aim of this library is to provide an easy way to write multi-channel image stac

23 Dec 27, 2022
Powerful Python library for atomic file writes.

Powerful Python library for atomic file writes.

Markus Unterwaditzer 313 Oct 19, 2022
Python's Filesystem abstraction layer

PyFilesystem2 Python's Filesystem abstraction layer. Documentation Wiki API Documentation GitHub Repository Blog Introduction Think of PyFilesystem's

pyFilesystem 1.8k Jan 02, 2023
This python project contains a class FileProcessor which allows one to grab a file and get some meta data and header information from it

This python project contains a class FileProcessor which allows one to grab a file and get some meta data and header information from it. In the current state, it outputs a PrettyTable to txt file as

Joshua Wren 1 Nov 09, 2021
Lumar - Smart File Creator

Lumar is a free tool for creating and managing files. With Lumar you can quickly create any type of file, add a file content and file size. With Lumar you can also find out if Photoshop or other imag

Paul - FloatDesign 3 Dec 10, 2021
A simple Python code that takes input from a csv file and makes it into a vcf file.

Contacts-Maker A simple Python code that takes input from a csv file and makes it into a vcf file. Imagine a college or a large community where each y

1 Feb 13, 2022
Better directory iterator and faster os.walk(), now in the Python 3.5 stdlib

scandir, a better directory iterator and faster os.walk() scandir() is a directory iteration function like os.listdir(), except that instead of return

Ben Hoyt 506 Dec 29, 2022
Pure Python tools for reading and writing all TIFF IFDs, sub-IFDs, and tags.

Tiff Tools Pure Python tools for reading and writing all TIFF IFDs, sub-IFDs, and tags. Developed by Kitware, Inc. with funding from The National Canc

Digital Slide Archive 32 Dec 14, 2022
Generates a clean .txt file of contents of a 3 lined csv file

Generates a clean .txt file of contents of a 3 lined csv file. File contents is the .gml file of some function which stores the contents of the csv as a map.

Alex Eckardt 1 Jan 09, 2022
Some-tasks - Files for some of the tasks for the group sessions

Files for some of the tasks for the group sessions Here you can find some of the

<a href=[email protected] Computer Networks"> 0 Aug 25, 2022
Instant Fuzzy File Search for Alfred

List all the files inside a folder using fd, and instantly fuzzy-search through all of them using fzf, all from inside Alfred with a single keyword: fzf.

Mr. Pennyworth 37 Nov 30, 2022
Extract an archive file (zip file or tar file) stored on AWS S3

S3 Extract Extract an archive file (zip file or tar file) stored on AWS S3. Details Downloads archive from S3 into memory, then extract and re-upload

Evan 1 Dec 14, 2021
Object-oriented file system path manipulation

path (aka path pie, formerly path.py) implements path objects as first-class entities, allowing common operations on files to be invoked on those path

Jason R. Coombs 1k Dec 28, 2022
BREP : Binary Search in plaintext and gzip files

BREP : Binary Search in plaintext and gzip files Search large files in O(log n) time using binary search. We support plaintext and Gzipped files. Benc

Arnaud de Saint Meloir 5 Dec 24, 2021
QSynthesis is a Python3 API to perform I/O based program synthesis of bitvector expressions.

QSynthesis is a Python3 API to perform I/O based program synthesis of bitvector expressions. It aims at facilitating code deobfuscation. The algorithm is greybox approach combining both a blackbox I/

Quarkslab 103 Dec 30, 2022
Convert All TXT Files To One File.

AllToOne Convert All TXT Files To One File. Hi šŸ‘‹ , I'm Alireza A Python Developer Boy šŸ”­ I’m currently working on my C# projects 🌱 I’m currently Lea

4 Jun 07, 2022
Convert CSV files into a SQLite database

csvs-to-sqlite Convert CSV files into a SQLite database. Browse and publish that SQLite database with Datasette. Basic usage: csvs-to-sqlite myfile.cs

Simon Willison 731 Dec 27, 2022