Hi! Thank you for publishing your code publicly.

I am a researcher who works with many tools that identify phage in metagenomes. However, I like to be confident in the implementation of the concepts. I noticed that your repository does not have any formal testing. Without tests, I am always skeptical about implementing a tool in my own work because I cannot be sure it is working as described.

Would your team be interested in adding tests to the code (e.g. using pytest)? If I, or another developer, were to create a pull request that implemented testing, would your team consider accepting such a request?

Also, it is a small thing, but I noticed that your code is not formatted in any community-accepted way. Would you consider accepting a pull request that has passed the code through a linter such as yapf or black? I usually add linting as part of my test suites.

Hi Kenneth,

preprocessing.py is a pretty generic name; maybe rename the script to PhaMer_preprocess.py to avoid potential future conflicts with other software?

Any plans on creating a bioconda recipe for PhaMer? That would greatly help users with the install & version management of PhaMer.

Also in regards to:

Because the parameter is larger than 100M, please make sure you have downloaded transformer.pth correctly.

Why not just use md5sum?

Hi,

• introduce ---threads to control threading behavior
• allow to supply external database directory, so DIAMOND database formatting isn't needed every time
• use 'pprodigal' for faster gene prediction step
• removed unused imports

Please note I didn't yet add pprodigal to the conda yaml - feel free to do so if you want to include it

Hello,

It seems that this repository is exceeding its data transfer limits. I believe you are aware of this, as you instruct users to download the transformer.pth from Google Drive.

However, it seems that I cannot clone the repository in general. I just want to make sure this is not a problem on my end, so I will walk through what I am doing.

Reproducible Example

First, cloning the repository

$ git clone [email protected]:KennthShang/PhaMer.git
Cloning into 'PhaMer'...
Downloading database/transformer.pth (143 MB)
Error downloading object: database/transformer.pth (28a82c1): Smudge error: Error downloading database/transformer.pth (28a82c1ca0fb2499c0071c685dbf49f3a0d060fdc231bb04f7535e88e7fe0858): batch response: This repository is over its data quota. Account responsible for LFS bandwidth should purchase more data packs to restore access.

Errors logged to /xdisk/bhurwitz/mig2020/rsgrps/bhurwitz/schackartk/projects/PhaMer/.git/lfs/logs/20220124T113548.16324098.log
Use `git lfs logs last` to view the log.
error: external filter git-lfs smudge -- %f failed 2
error: external filter git-lfs smudge -- %f failed
fatal: database/transformer.pth: smudge filter lfs failed
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry the checkout with 'git checkout -f HEAD'

Checking git status as suggested indicates that several files were not checked out

$ cd PhaMer/
$ git status
# On branch main
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#	deleted:    .gitattributes
#	deleted:    LICENSE.txt
#	deleted:    PhaMer.py
#	deleted:    PhaMer.yaml
#	deleted:    README.md
#	deleted:    database/.DS_Store
#	deleted:    database/contigs.csv
#	deleted:    database/database.fa.bz2
#	deleted:    database/pc2wordsid.dict
#	deleted:    database/pcs.csv
#	deleted:    database/profiles.csv
#	deleted:    database/proteins.csv
#	deleted:    database/transformer.pth
#	deleted:    logo.jpg
#	deleted:    model.py
#	deleted:    preprocessing.py
#	deleted:    test_contigs.fa
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#	.gitattributes
#	LICENSE.txt
#	PhaMer.py
#	PhaMer.yaml
#	README.md
#	database/

To confirm that several files are missing, such as preprocessing.py.

$ ls
database  LICENSE.txt  PhaMer.py  PhaMer.yaml  README.md

Continuing with installation instructions anyway in case they resolve these issues.

$ conda env create -f PhaMer.yaml -n phamer
Collecting package metadata (repodata.json): done
Solving environment: done
Preparing transaction: done
Verifying transaction: done
Executing transaction: \ By downloading and using the CUDA Toolkit conda packages, you accept the terms and conditions of the CUDA End User License Agreement (EULA): https://docs.nvidia.com/cuda/eula/index.html

done
Installing pip dependencies: / Ran pip subprocess with arguments:
['/home/u29/schackartk/.conda/envs/phamer/bin/python', '-m', 'pip', 'install', '-U', '-r', '/xdisk/bhurwitz/mig2020/rsgrps/bhurwitz/schackartk/projects/PhaMer/condaenv.1nhq2jy1.requirements.txt']
Pip subprocess output:
Collecting joblib==1.1.0
  Using cached joblib-1.1.0-py2.py3-none-any.whl (306 kB)
Collecting scikit-learn==1.0.1
  Using cached scikit_learn-1.0.1-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (25.9 MB)
Collecting sklearn==0.0
  Using cached sklearn-0.0-py2.py3-none-any.whl
Collecting threadpoolctl==3.0.0
  Using cached threadpoolctl-3.0.0-py3-none-any.whl (14 kB)
Requirement already satisfied: scipy>=1.1.0 in /home/u29/schackartk/.conda/envs/phamer/lib/python3.8/site-packages (from scikit-learn==1.0.1->-r /xdisk/bhurwitz/mig2020/rsgrps/bhurwitz/schackartk/projects/PhaMer/condaenv.1nhq2jy1.requirements.txt (line 2)) (1.7.1)
Requirement already satisfied: numpy>=1.14.6 in /home/u29/schackartk/.conda/envs/phamer/lib/python3.8/site-packages (from scikit-learn==1.0.1->-r /xdisk/bhurwitz/mig2020/rsgrps/bhurwitz/schackartk/projects/PhaMer/condaenv.1nhq2jy1.requirements.txt (line 2)) (1.21.2)
Installing collected packages: threadpoolctl, joblib, scikit-learn, sklearn
Successfully installed joblib-1.1.0 scikit-learn-1.0.1 sklearn-0.0 threadpoolctl-3.0.0

done
#
# To activate this environment, use
#
#     $ conda activate phamer
#
# To deactivate an active environment, use
#
#     $ conda deactivate

Activating the conda environment and attempting to get transformer.pth

$ conda activate phamer
$ cd database
$ bzip2 -d database.fa.bz2
$ git lfs install
Updated git hooks.
Git LFS initialized.
$ rm transformer.pth
rm: cannot remove ‘transformer.pth’: No such file or directory
$ git checkout .
error: pathspec './' did not match any file(s) known to git.

$ ls
contigs.csv  database.fa  pc2wordsid.dict  pcs.csv  profiles.csv  proteins.csv

Since the file doesn't seem to exist, I followed your Google Drive link and pasted into database/ manually.

$ ls
contigs.csv  database.fa  pc2wordsid.dict  pcs.csv  profiles.csv  proteins.csv  transformer.pth

I will try checking out again.

$ git checkout .
error: pathspec './' did not match any file(s) known to git.

Going back up, you can see that I am still missing scripts.

$ cd ..
$ ls
metaphinder_reprex  phage_finders  PhaMer  snakemake_tutorial

Conclusions

I am missing the scripts and cannot run the tool. I believe this all comes down to the repo exceedingits data transfer limits. This is probably due to you storing the large database files in the repository.

Possible solution?

Going forward, maybe entirely remove the large files from the repo so that you don't exceed limits. I am not sure what I can do at this moment since the limits are already exceeded.

Also, it would be helpful if I could obtain the transformer.pth from the command line (e.g. using wget) since I, and many researchers, are working on an HPC or cloud.

Thank you, -Ken