Location of public benchmarking; primarily final results

Overview

CSL_public_benchmark

This repo is intended to provide a periodically-updated, public view into genome sequencing benchmarks managed by HudsonAlpha's Clinical Services Lab (CSL). The benchmarking results primarily provide the CSL a systematic approach to evalute various reference genome, aligner, and variant caller combinations against each other. All of the datasets we used for testing were generated at HudsonAlpha. The short-read PCR-free datasets were generated using standard clinical processes in the CSL and are currently private datasets. The long-read PacBio datasets were generated by the Genome Sequencing Center and are publicly hosted through the Genome in a Bottle consortium (see below).

The benchmarks or "truth sets" themselves are large-scale publicly available benchmarks created for a handful of reference samples. Most of the benchmarks we use were generated by the Genome in a Bottle (GIAB) Consortium.

Current status

This initial release just includes the final results files that are reviewed after the pipelines have completed.

What is the pipeline?

The benchmarking pipeline itself is maintained in a private repo. Briefly, it is a snakemake pipeline that built around a systematic final evaluation that mostly uses RTG vcfeval to measure sensitivity and precision. The primary "wildcards" in this evaluation are the reference, the aligner, and the variant caller; with versioning where appropriate. This allows us to quickly add new tools by defining new rules to run a particular tool (typically one per aligner or caller), and then evaluate in a standard way. In general, we try to use docker images or conda environments when these are already available to increase downstream portability; however, these are not always available.

As a result, many rules are tied to our cluster ecosystem, either through modules and/or file paths to installed software. Additionally, all the metadata (e.g. fastq pairs for a given sample) is tracked using an internal system. This means that this pipeline, even if publically available, would definitely not run "out-of-the-box" for anyone outside of HudsonAlpha. A very long-term goal would be to create a public version that can run out-of-the-box given user-provided metadata.

However, in the interest of transparency, we will be making efforts to clarify any questions about the implementation over time. This will largely be driven by questions we receive from the community (i.e. create issues if you have questions, so we can begin tackling this). Examples of things already on the TODO radar:

  • Rules for aligners and callers
  • Rules for evaluation
  • Description/links to specific reference files
Comments
  • truvari options

    truvari options

    Hi, Thanks for testing dysgu, I have a few questions about whats causing the low precision values of dysgu. Firstly, I just wanted to compare notes about how you test using truvari. I have tested with: grep '#\|PASS' HG002.dysgu.vcf | grep '#\|DEL' > HG002.dysgu_pass.del.vcf; bgzip HG002.dysgu_pass.del.vcf; tabix HG002.dysgu_pass.del.vcf; truvari bench -b HG002_SVs_Tier1_v0.6.vcf.gz --includebed HG002_SVs_Tier1_v0.6_include.bed --sizemax 260000000 --giabreport -c HG002.dysgu_pass.del.vcf.gz -o truvari_dysgu_pass_only_del --pctsim 0 -s 50 --passonly Also was interested to know what coverage and read length your samples are. Finally I noticed in the dysgu results the stdev of the precision values seemed very high, for example on the hg38 GIAB masked dragmap-1.2.1 precision score was 0.4321+-0.2749. Im not sure whats causing this, but possibly the insert size metrics are not being worked out properly? This should be available in the log file from dysgu. Thanks!

    opened by kcleal 10
  • Adds the tandem-repeat option to sniffles

    Adds the tandem-repeat option to sniffles

    • This adds the tandem-repeat option under temporary name "sniffles_tr-{version}"; this will eventually replace "sniffles-{version}"
    opened by holtjma 2
  • Deletion split

    Deletion split

    • Splits deletion benchmark into RESTRICTED (requiring a high-confidence BED file) and UNRESTRICTED (no BED files, it's everything in the benchmark VCF)
    • RESTRICTED now only contains HG002
    • UNRESTRICTED has both HG001 and HG002, but with caveats around precision
    • Sniffles v2.0.2 happened to drop while I was testing this, so I went ahead and added it (it ran quite fast)
    opened by holtjma 2
  • Benchmark metadata

    Benchmark metadata

    • Adds a benchmarks folder to describe where benchmark files came from
    • Small variants and CMRG both have a simple shell script for downloading the exact files used
    • Deletion files are originally in hg19, and a semi-manual process was used to liftover the files to hg38. The steps for this process are described in the README and the final VCF files are stored in this repo due to their relatively small size.
    opened by holtjma 1
  • References metadata

    References metadata

    Primarily adds reference metadata for the two primary reference files we are currently testing with:

    • hg38_asm5_alt - includes scripts for download, creating ALT contigs, and the final alt files
    • hg38_GIAB_masked - includes scripts for download and the dummy alt file
    opened by holtjma 1
  • Add more information regarding the references

    Add more information regarding the references

    We've had some questions about the reference origins. We should probably add some links to file and bash scripts where applicable with regards to reference acquisition.

    documentation 
    opened by holtjma 1
  • Release 20220225

    Release 20220225

    • Adds hg38_T2T_masked reference genome and metadata around generated corresponding reference files
    • Benchmark results adds hg38_T2T_masked results as well
    • On the backend, Truvari was updated to v3.1.0, this did not seem to have a significant impact on results
    opened by holtjma 0
  • adding results files

    adding results files

    • Added haplotyping results for caller cyrius
    • Updated versions of pbmm2 and pbsv; there are some changes associated with the results of these, so the previous version is maintained for this release
    opened by holtjma 0
Releases(2022-06-10)
  • 2022-06-10(Jun 10, 2022)

  • 2022-05-19(May 19, 2022)

    This release is primarily adding two new samples to the PCR-free datasets. The following updates occurred as a result of this change:

    • Adds two new samples to our PCR-free datasets corresponding to HG006 and HG007
    • PCR-free results all shifted slightly (the vast majority to slightly worse performance); we did not notice any drastic changes across the results; all average sensitivities, precisions, and F1-scores shifted <0.001
    • Updates our expected CYP2D6 outputs to include expectations for HG006 and HG007
    Source code(tar.gz)
    Source code(zip)
    results_20220519.pdf(143.74 KB)
    small_summary_20220519.csv(5.36 KB)
  • 2022-04-29(Apr 29, 2022)

  • 2022-04-08(Apr 8, 2022)

  • 2022-03-18(Mar 18, 2022)

  • 2022-02-25(Feb 25, 2022)

    Reference changes:

    • Adds the hg38_T2T_masked reference which is version 2 of the hg38_GIAB_masked reference. A brief description and direct download links are provided with the reference metadata.
    • The hg38_T2T_masked results tend to be very slightly better than the v1 results, so hg38_GIAB_masked will likely be retired in a future release.

    Method changes:

    Source code(tar.gz)
    Source code(zip)
    results_20220225.pdf(164.79 KB)
    small_summary_20220225.csv(8.97 KB)
  • 2022-02-18(Feb 18, 2022)

  • 2022-02-11(Feb 11, 2022)

    Software changes:

    • Added dnascope-1.0-202112.01-PO for PCR-free datasets, dnascope-0.5-202010.04-PO will be removed in future releases. Additionally, the pass-only filter (e.g. -PO) is recommended for DNAscope, so the unfiltered version has been remove from reporting. Thanks to @DonFreed for the recommendations!
    • Added dysgu-1.3.4-PO, which is a pass-only filtered version of dysgu-1.3.4, for PCR-free and PacBio datasets. Additionally, the pass-only filter (e.g. -PO) is recommended for dysgu, so the unfiltered version will be removed in future released. Thanks to @kcleal for the recommendations!

    Other changes:

    • Added a note in the README on release cadence. In order to reduce overhead, going forward we will limit formal releases to at most once a week. New or partial results may appear through the week with the intention to summarize any changes in the weekly release.
    Source code(tar.gz)
    Source code(zip)
    results_20220211.pdf(157.26 KB)
    small_summary_20220211.csv(8.80 KB)
  • 2022-02-09(Feb 9, 2022)

  • 2022-02-08(Feb 8, 2022)

    Method change:

    • Splits deletion benchmark into RESTRICTED (requiring a high-confidence BED file) and UNRESTRICTED (no BED files, it's everything in the benchmark VCF)
    • RESTRICTED now only contains HG002 with the Tier1 regions. This set is more fair when judging the precision of the aligner/caller pair.
    • UNRESTRICTED has both HG001 and HG002, but with caveats around precision. This set includes more total variants and two samples, but precision is less accurate.

    Software addition:

    • Sniffles v2.0.2 was added as a new caller
    Source code(tar.gz)
    Source code(zip)
    results_20220208.pdf(153.97 KB)
    small_summary_20220208.csv(8.83 KB)
  • 2022-02-07(Feb 7, 2022)

  • 2022-01-28(Jan 28, 2022)

    • Added the first haplotyping caller to our results with cyrius-1.1.1; note that this caller is designed to work on short-read datasets and the upstream tooling (both reference and aligner) can have significant impact on its performance
    • Updated versions of pbmm2 (1.4.0 -> 1.7.0) and pbsv (2.6.2 -> 2.8.0); there are some changes in performance between the previous versions so they are retained in this release; they will be removed in subsequent releases
    Source code(tar.gz)
    Source code(zip)
  • 2022-01-07(Jan 7, 2022)

    Two variant caller updates:

    • Clair3 was update to v0.1-r9: Our previous version was v0.1-r5, and it was running in a conda environment after some back-and-forth with the developers. They now have a docker image that is much easier to use, so we have switched to that for both the Illumina and PacBio tests.
    • PEPPER-Margin-DeepVariant was added a full caller on version r0.7: Previously, we were treating this process as a BAM modifier (basically for phasing) and ignoring any variant calling results. With this change, it now operates as a variant caller and the VCF is analyzed with the rest of the callers. We are using the developer-released docker image for our analysis.
    • We have removed old versions of both tools to avoid any confusion around the analysis implementation
    Source code(tar.gz)
    Source code(zip)
    results_20220107.pdf(145.82 KB)
  • 2021-12-17(Dec 17, 2021)

    Two main results changes:

    • Adds the dnascope-0.5-202010.04-PO variant caller: This is the same data as dnascope-0.5-202010.04 but with a PASS-only filter applied to the VCF file. The short-read DNAscope callers uses the FILTER field to annotate variants that are rejected by the model as likely false positives. This has significant impact on the results and is recommended as best practice by the developers. Thanks to @DonFreed for helping diagnose the issue!
    • Updates minimap2 aligner from v2.22 to v2.23: Overall, this had minimal impact in our benchmark. v2.22 will be retired from the benchmark next release.
    Source code(tar.gz)
    Source code(zip)
    results_20211217.pdf(145.62 KB)
  • 2021-12-13(Dec 13, 2021)

    Metadata changes:

    • Adds a references folder for tracking references that are used in the analysis
    • Adds the hg38_asm5_alt reference including links to the reference and a script demonstrating how the ALT contigs were remapped
    • Adds the hg38_GIAB_masked reference including links to the reference and a dummy ALT file used for the pipeline

    Results changes:

    • Added the SNAP-2.0.0 caller that was recently released, this was run with the -hc- option so GATK-based results are expected to not be as accurate
    Source code(tar.gz)
    Source code(zip)
    results_20211213.pdf(140.63 KB)
Owner
HudsonAlpha Institute for Biotechnology
HudsonAlpha Institute for Biotechnology
Grimoire is a Python library for creating interactive fiction as hyperlinked html.

Grimoire Grimoire is a Python library for creating interactive fiction as hyperlinked html. Installation pip install grimoire-if Usage Check out the

Scott Russell 5 Oct 11, 2022
Ningyu Jia(nj2459)/Mengyin Ma(mm5937) Call Analysis group project(Group 36)

Group and Section Group 36 Section 001 name and UNI Name UNI Ningyu Jia nj2459 Mengyin Ma mm5937 code explanation Parking.py (1) Calculate the rate of

1 Dec 04, 2021
Hello, Welcome to this repo. don't forget to read guidelines in readme.md

Hacktoberfest_2021 If you looking for your first contribution, we are here to help. Just create a simple program using any language you like in our fo

Wafa Rifqi Anafin 117 Dec 14, 2022
Stori QA Automation Challenge

Stori-QA-Automation-Challenge This is the repository is created for the Stori QA Intern Automation Engineer Challenge! In this you can find the Requir

Daniel CastaΓ±eda 0 Feb 20, 2022
Python bindings for `ign-msgs` and `ign-transport`

Python Ignition This project aims to provide Python bindings for ignition-msgs and ignition-transport. It is a work in progress... C++ and Python libr

Rhys Mainwaring 3 Nov 08, 2022
Collection of tools to be more productive in your work environment and to avoid certain repetitive tasks. πŸ’›πŸ’™πŸ’š

Collection of tools to be more productive in your work environment and to avoid certain repetitive tasks. πŸ’›πŸ’™πŸ’š

Raja Rakotonirina 2 Jan 10, 2022
A repository of study materials related to Think Python 2nd Edition by Allen B. Downey. More information about the book can be found here: https://greenteapress.com/wp/think-python-2e/

Intro-To-Python This content is based on the book Think Python 2nd Edition by Allen B. Downey. More information about the book can be found here: http

Brent Eskridge 63 Jan 07, 2023
Just some information about this nerd.

Greetings, mates, I am ErrorDIM - aka ErrorDimension πŸ‘‹ 🧬 Programming Languages I Can Use: πŸ₯‡ Top Starred Repositories: # Name Stars Size Major Langu

ErrorDIM 3 Jan 11, 2022
Your copilot to studies and work (Pomodoro-timer, Translate and Notes app)

Copylot Your copilot to studies and work (Pomodoro-timer, Translate and Notes app) Copylot are three applications in one: Pomodoro Translate Notes Cop

Eduardo Mendes 20 Dec 16, 2022
More routines for operating on iterables, beyond itertools

More Itertools Python's itertools library is a gem - you can compose elegant solutions for a variety of problems with the functions it provides. In mo

2.8k Jan 02, 2023
Contains the code of my learning of Python OOP.

OOP Python This repository contains the code of my learning of Python OOP. All the code: is following PEP 8 βœ… has proper concept illustrations and com

Samyak Jain 2 Jan 15, 2022
Collection of functions for working with interlaced content in VapourSynth.

vsfieldkit Collection of functions for working with interlaced content in VapourSynth. It does not have any hard dependencies outside of VapourSynth.

Justin Turner Arthur 11 May 27, 2022
Basic Clojure REPL for Sublime Text

Basic Clojure REPL for Sublime Text Goals: Decomplected: just REPL, nothing more Zero dependencies: works directly with pREPL Compact: Display code ev

Nikita Prokopov 23 Dec 24, 2021
An Airdrop alternative for cross-platform users only for desktop with Python

PyDrop An Airdrop alternative for cross-platform users only for desktop with Python, -version 1.0 with less effort, just as a practice. ##############

Bernardo Olisan 6 Mar 25, 2022
Rick Astley Language is a rick roll oriented, dynamic, strong, esoteric programming language.

Rick Roll Language / Rick Astley Language A rick roll oriented, dynamic, strong, esoteric programming language. Prolegomenon The reasons that I made t

Rick Roll Programming Language 658 Jan 09, 2023
InfiniPy has some neat features - like the endpoint for function

InfiniPy has some neat features - like the endpoint for function

ZeroTwo 7 Nov 20, 2022
Problem 5: Fermat near-misses

Problem 5: Fermat near-misses fermatnearmiss This is a script that computes fermat nearm misses when the -f option is set and requires users to input

CHRIS BYRON (Int0x80) 1 Jan 08, 2022
Notes on the Deep Learning book from Ian Goodfellow, Yoshua Bengio and Aaron Courville (2016)

The Deep Learning Book - Goodfellow, I., Bengio, Y., and Courville, A. (2016) This content is part of a series following the chapter 2 on linear algeb

hadrienj 1.7k Jan 07, 2023
Metal Gear Rising: Revengeance's DAT archive (un)packer

DOOMP Metal Gear Rising: Revengeance's DAT archive (un)packer

Christopher Holzmann PΓ©rez 5 Sep 02, 2022
A modern python module including many useful features that make discord bot programming extremely easy.

discord-super-utils Documentation Secondary Documentation A modern python module including many useful features that make discord bot programming extr

106 Dec 19, 2022