Accurate identification of bacteriophages from metagenomic data using Transformer

Related tags

Deep LearningPhaMer
Overview

PhaMer

PhaMer is a python library for identifying bacteriophages from metagenomic data. PhaMer is based on a Transorfer model and rely on protein-based vocabulary to convert DNA sequences into sentences.

Overview

The main function of PhaMer is to identify phage-like contigs from metagenomic data. The input of the program should be fasta files and the output will be a csv file showing the predictions. Since it is a Deep learning model, if you have GPU units on your PC, we recommand you to use them to save your time.

If you have any trouble installing or using PhaMer, please let us know by opening an issue on GitHub or emailing us ([email protected]).

Required Dependencies

If you want to use the gpu to accelerate the program:

  • cuda

  • Pytorch-gpu

  • For cpu version pytorch: conda install pytorch torchvision torchaudio cpuonly -c pytorch

  • For gpu version pytorch: Search pytorch to find the correct cuda version according to your computer

An easiler way to install

Note: we suggest you to install all the package using conda (both miniconda and Anaconda are ok).

After cloning this respository, you can use anaconda to install the PhaMer.yaml. This will install all packages you need with gpu mode (make sure you have installed cuda on your system to use the gpu version. Othervise, it will run with cpu version). The command is: conda env create -f PhaMer.yaml -n phamer

Prepare the database and environment

Due to the limited size of the GitHub, we zip the database. Before using PhaMer, you need to unpack them using the following commands.

  1. When you use PhaMer at the first time
cd PhaMer/
conda env create -f PhaMer.yaml -n phamer
conda activate phamer
cd database/
bzip2 -d database.fa.bz2
git lfs install
rm transformer.pth
git checkout .
cd ..

Note: Because the parameter is larger than 100M, please make sure you have installed git-lfs to downloaded it from GitHub

  1. If the example can be run without any but bugs, you only need to activate your 'phamer' environment before using PhaMer.
conda activate phamer

Usage

python preprocessing.py [--contigs INPUT_FA] [--len MINIMUM_LEN]
python PhaMer.py [--out OUTPUT_CSV] [--reject THRESHOLD]

Options

  --contigs INPUT_FA
                        input fasta file
  --len MINIMUM_LEN
                        predict only for sequence >= len bp (default 3000)
  --out OUTPUT_CSV
                        The output csv file (prediction)
  --reject THRESHOLD
                        Threshold to reject prophage. The higher the value, the more prophage will be rejected (default 0.3)

Example

Prediction on the example file:

python preprocessing.py --contigs test_contigs.fa
python PhaMer.py --out example_prediction.csv

The prediction will be written in example_prediction.csv. The CSV file has three columns: contigs names, prediction, and prediction score.

References

The paper is submitted to the ISMB 2022.

The arXiv version can be found via: Accurate identification of bacteriophages from metagenomic data using Transformer

Contact

If you have any questions, please email us: [email protected]

Comments
  • issues collections from schackartk (solved)

    issues collections from schackartk (solved)

    Hi! Thank you for publishing your code publicly.

    I am a researcher who works with many tools that identify phage in metagenomes. However, I like to be confident in the implementation of the concepts. I noticed that your repository does not have any formal testing. Without tests, I am always skeptical about implementing a tool in my own work because I cannot be sure it is working as described.

    Would your team be interested in adding tests to the code (e.g. using pytest)? If I, or another developer, were to create a pull request that implemented testing, would your team consider accepting such a request?

    Also, it is a small thing, but I noticed that your code is not formatted in any community-accepted way. Would you consider accepting a pull request that has passed the code through a linter such as yapf or black? I usually add linting as part of my test suites.

    opened by schackartk 12
  • Rename preprocessing.py?

    Rename preprocessing.py?

    Hi Kenneth,

    preprocessing.py is a pretty generic name; maybe rename the script to PhaMer_preprocess.py to avoid potential future conflicts with other software?

    opened by sjaenick 1
  • bioconda recipe

    bioconda recipe

    Any plans on creating a bioconda recipe for PhaMer? That would greatly help users with the install & version management of PhaMer.

    Also in regards to:

    Because the parameter is larger than 100M, please make sure you have downloaded transformer.pth correctly.

    Why not just use md5sum?

    opened by nick-youngblut 1
  • Threading/Performance updates

    Threading/Performance updates

    Hi,

    • introduce ---threads to control threading behavior
    • allow to supply external database directory, so DIAMOND database formatting isn't needed every time
    • use 'pprodigal' for faster gene prediction step
    • removed unused imports

    Please note I didn't yet add pprodigal to the conda yaml - feel free to do so if you want to include it

    opened by sjaenick 1
  • Bug: Unable to clone repository

    Bug: Unable to clone repository

    Hello,

    It seems that this repository is exceeding its data transfer limits. I believe you are aware of this, as you instruct users to download the transformer.pth from Google Drive.

    However, it seems that I cannot clone the repository in general. I just want to make sure this is not a problem on my end, so I will walk through what I am doing.

    Reproducible Example

    First, cloning the repository

    $ git clone [email protected]:KennthShang/PhaMer.git
    Cloning into 'PhaMer'...
    Downloading database/transformer.pth (143 MB)
    Error downloading object: database/transformer.pth (28a82c1): Smudge error: Error downloading database/transformer.pth (28a82c1ca0fb2499c0071c685dbf49f3a0d060fdc231bb04f7535e88e7fe0858): batch response: This repository is over its data quota. Account responsible for LFS bandwidth should purchase more data packs to restore access.
    
    Errors logged to /xdisk/bhurwitz/mig2020/rsgrps/bhurwitz/schackartk/projects/PhaMer/.git/lfs/logs/20220124T113548.16324098.log
    Use `git lfs logs last` to view the log.
    error: external filter git-lfs smudge -- %f failed 2
    error: external filter git-lfs smudge -- %f failed
    fatal: database/transformer.pth: smudge filter lfs failed
    warning: Clone succeeded, but checkout failed.
    You can inspect what was checked out with 'git status'
    and retry the checkout with 'git checkout -f HEAD'
    

    Checking git status as suggested indicates that several files were not checked out

    $ cd PhaMer/
    $ git status
    # On branch main
    # Changes to be committed:
    #   (use "git reset HEAD <file>..." to unstage)
    #
    #	deleted:    .gitattributes
    #	deleted:    LICENSE.txt
    #	deleted:    PhaMer.py
    #	deleted:    PhaMer.yaml
    #	deleted:    README.md
    #	deleted:    database/.DS_Store
    #	deleted:    database/contigs.csv
    #	deleted:    database/database.fa.bz2
    #	deleted:    database/pc2wordsid.dict
    #	deleted:    database/pcs.csv
    #	deleted:    database/profiles.csv
    #	deleted:    database/proteins.csv
    #	deleted:    database/transformer.pth
    #	deleted:    logo.jpg
    #	deleted:    model.py
    #	deleted:    preprocessing.py
    #	deleted:    test_contigs.fa
    #
    # Untracked files:
    #   (use "git add <file>..." to include in what will be committed)
    #
    #	.gitattributes
    #	LICENSE.txt
    #	PhaMer.py
    #	PhaMer.yaml
    #	README.md
    #	database/
    

    To confirm that several files are missing, such as preprocessing.py.

    $ ls
    database  LICENSE.txt  PhaMer.py  PhaMer.yaml  README.md
    

    Continuing with installation instructions anyway in case they resolve these issues.

    $ conda env create -f PhaMer.yaml -n phamer
    Collecting package metadata (repodata.json): done
    Solving environment: done
    Preparing transaction: done
    Verifying transaction: done
    Executing transaction: \ By downloading and using the CUDA Toolkit conda packages, you accept the terms and conditions of the CUDA End User License Agreement (EULA): https://docs.nvidia.com/cuda/eula/index.html
    
    done
    Installing pip dependencies: / Ran pip subprocess with arguments:
    ['/home/u29/schackartk/.conda/envs/phamer/bin/python', '-m', 'pip', 'install', '-U', '-r', '/xdisk/bhurwitz/mig2020/rsgrps/bhurwitz/schackartk/projects/PhaMer/condaenv.1nhq2jy1.requirements.txt']
    Pip subprocess output:
    Collecting joblib==1.1.0
      Using cached joblib-1.1.0-py2.py3-none-any.whl (306 kB)
    Collecting scikit-learn==1.0.1
      Using cached scikit_learn-1.0.1-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (25.9 MB)
    Collecting sklearn==0.0
      Using cached sklearn-0.0-py2.py3-none-any.whl
    Collecting threadpoolctl==3.0.0
      Using cached threadpoolctl-3.0.0-py3-none-any.whl (14 kB)
    Requirement already satisfied: scipy>=1.1.0 in /home/u29/schackartk/.conda/envs/phamer/lib/python3.8/site-packages (from scikit-learn==1.0.1->-r /xdisk/bhurwitz/mig2020/rsgrps/bhurwitz/schackartk/projects/PhaMer/condaenv.1nhq2jy1.requirements.txt (line 2)) (1.7.1)
    Requirement already satisfied: numpy>=1.14.6 in /home/u29/schackartk/.conda/envs/phamer/lib/python3.8/site-packages (from scikit-learn==1.0.1->-r /xdisk/bhurwitz/mig2020/rsgrps/bhurwitz/schackartk/projects/PhaMer/condaenv.1nhq2jy1.requirements.txt (line 2)) (1.21.2)
    Installing collected packages: threadpoolctl, joblib, scikit-learn, sklearn
    Successfully installed joblib-1.1.0 scikit-learn-1.0.1 sklearn-0.0 threadpoolctl-3.0.0
    
    done
    #
    # To activate this environment, use
    #
    #     $ conda activate phamer
    #
    # To deactivate an active environment, use
    #
    #     $ conda deactivate
    

    Activating the conda environment and attempting to get transformer.pth

    $ conda activate phamer
    $ cd database
    $ bzip2 -d database.fa.bz2
    $ git lfs install
    Updated git hooks.
    Git LFS initialized.
    $ rm transformer.pth
    rm: cannot remove ‘transformer.pth’: No such file or directory
    $ git checkout .
    error: pathspec './' did not match any file(s) known to git.
    
    $ ls
    contigs.csv  database.fa  pc2wordsid.dict  pcs.csv  profiles.csv  proteins.csv
    

    Since the file doesn't seem to exist, I followed your Google Drive link and pasted into database/ manually.

    $ ls
    contigs.csv  database.fa  pc2wordsid.dict  pcs.csv  profiles.csv  proteins.csv  transformer.pth
    

    I will try checking out again.

    $ git checkout .
    error: pathspec './' did not match any file(s) known to git.
    

    Going back up, you can see that I am still missing scripts.

    $ cd ..
    $ ls
    metaphinder_reprex  phage_finders  PhaMer  snakemake_tutorial
    

    Conclusions

    I am missing the scripts and cannot run the tool. I believe this all comes down to the repo exceedingits data transfer limits. This is probably due to you storing the large database files in the repository.

    Possible solution?

    Going forward, maybe entirely remove the large files from the repo so that you don't exceed limits. I am not sure what I can do at this moment since the limits are already exceeded.

    Also, it would be helpful if I could obtain the transformer.pth from the command line (e.g. using wget) since I, and many researchers, are working on an HPC or cloud.

    Thank you, -Ken

    opened by schackartk 1
Releases(v1.0)
Owner
Kenneth Shang
Kenneth Shang
Parallel and High-Fidelity Text-to-Lip Generation; AAAI 2022 ; Official code

Parallel and High-Fidelity Text-to-Lip Generation This repository is the official PyTorch implementation of our AAAI-2022 paper, in which we propose P

Zhying 77 Dec 21, 2022
Library extending Jupyter notebooks to integrate with Apache TinkerPop and RDF SPARQL.

Graph Notebook: easily query and visualize graphs The graph notebook provides an easy way to interact with graph databases using Jupyter notebooks. Us

Amazon Web Services 501 Dec 28, 2022
SOTA easy to use PyTorch-based DL training library

Easily train or fine-tune SOTA computer vision models from one training repository. SuperGradients Introduction Welcome to SuperGradients, a free open

619 Jan 03, 2023
Code associated with the paper "Deep Optics for Single-shot High-dynamic-range Imaging"

Deep Optics for Single-shot High-dynamic-range Imaging Code associated with the paper "Deep Optics for Single-shot High-dynamic-range Imaging" CVPR, 2

Stanford Computational Imaging Lab 40 Dec 12, 2022
Unsupervised Learning of Video Representations using LSTMs

Unsupervised Learning of Video Representations using LSTMs Code for paper Unsupervised Learning of Video Representations using LSTMs by Nitish Srivast

Elman Mansimov 341 Dec 20, 2022
The implementation for "Comprehensive Knowledge Distillation with Causal Intervention".

Comprehensive Knowledge Distillation with Causal Intervention This repository is a PyTorch implementation of "Comprehensive Knowledge Distillation wit

Xiang Deng 10 Nov 03, 2022
This repository stores the code to reproduce the results published in "TiWS-iForest: Isolation Forest in Weakly Supervised and Tiny ML scenarios"

TinyWeaklyIsolationForest This repository stores the code to reproduce the results published in "TiWS-iForest: Isolation Forest in Weakly Supervised a

2 Mar 21, 2022
The Body Part Regression (BPR) model translates the anatomy in a radiologic volume into a machine-interpretable form.

Copyright © German Cancer Research Center (DKFZ), Division of Medical Image Computing (MIC). Please make sure that your usage of this code is in compl

MIC-DKFZ 40 Dec 18, 2022
Implementation of the paper NAST: Non-Autoregressive Spatial-Temporal Transformer for Time Series Forecasting.

Non-AR Spatial-Temporal Transformer Introduction Implementation of the paper NAST: Non-Autoregressive Spatial-Temporal Transformer for Time Series For

Chen Kai 66 Nov 28, 2022
DANA paper supplementary materials

DANA Supplements This repository stores the data, results, and R scripts to generate these reuslts and figures for the corresponding paper Depth Norma

0 Dec 17, 2021
Lucid library adapted for PyTorch

Lucent PyTorch + Lucid = Lucent The wonderful Lucid library adapted for the wonderful PyTorch! Lucent is not affiliated with Lucid or OpenAI's Clarity

Lim Swee Kiat 520 Dec 26, 2022
Just playing with getting VQGAN+CLIP running locally, rather than having to use colab.

Just playing with getting VQGAN+CLIP running locally, rather than having to use colab.

Nerdy Rodent 2.3k Jan 04, 2023
This project provides an unsupervised framework for mining and tagging quality phrases on text corpora with pretrained language models (KDD'21).

UCPhrase: Unsupervised Context-aware Quality Phrase Tagging To appear on KDD'21...[pdf] This project provides an unsupervised framework for mining and

Xiaotao Gu 146 Dec 22, 2022
How to Predict Stock Prices Easily Demo

How-to-Predict-Stock-Prices-Easily-Demo How to Predict Stock Prices Easily - Intro to Deep Learning #7 by Siraj Raval on Youtube ##Overview This is th

Siraj Raval 752 Nov 16, 2022
Source code for the BMVC-2021 paper "SimReg: Regression as a Simple Yet Effective Tool for Self-supervised Knowledge Distillation".

SimReg: A Simple Regression Based Framework for Self-supervised Knowledge Distillation Source code for the paper "SimReg: Regression as a Simple Yet E

9 Oct 15, 2022
PyTorch implementation of Barlow Twins.

Barlow Twins: Self-Supervised Learning via Redundancy Reduction PyTorch implementation of Barlow Twins. @article{zbontar2021barlow, title={Barlow Tw

Facebook Research 839 Dec 29, 2022
Episodic-memory - Ego4D Episodic Memory Benchmark

Ego4D Episodic Memory Benchmark EGO4D is the world's largest egocentric (first p

3 Feb 18, 2022
object detection; robust detection; ACM MM21 grand challenge; Security AI Challenger Phase VII

赛题背景 在商品知识产权领域,知识产权体现为在线商品的设计和品牌。不幸的是,在每一天,存在着非法商户通过一些对抗手段干扰商标识别来逃避侵权,这带来了很高的知识产权风险和财务损失。为了促进先进的多媒体人工智能技术的发展,以保护企业来之不易的创作和想法免受恶意使用和剽窃,因此提出了鲁棒性标识检测挑战赛

65 Dec 22, 2022
SiT: Self-supervised vIsion Transformer

This repository contains the official PyTorch self-supervised pretraining, finetuning, and evaluation codes for SiT (Self-supervised image Transformer).

Sara Ahmed 275 Dec 28, 2022
Meaningful titles for tabs and PDF downloads! Also supports tab search.

arxiv-utils If you are a researcher that reads a lot on ArXiv, you'll benefit a lot from this web extension. Renames the title of PDF page to the pape

Johnson 174 Dec 20, 2022