BioThings API framework - Making high-performance API for biological annotation data

Last update: Jan 04, 2023

Overview

BioThings SDK

Quick Summary

BioThings SDK provides a Python-based toolkit to build high-performance data APIs (or web services) from a single data source or multiple data sources. It has the particular focus on building data APIs for biomedical-related entities, a.k.a "BioThings" (such as genes, genetic variants, drugs, chemicals, diseases, etc.).

Documentation about BioThings SDK can be found at https://docs.biothings.io

Introduction

What's BioThings?

We use "BioThings" to refer to objects of any biomedical entity-type represented in the biological knowledge space, such as genes, genetic variants, drugs, chemicals, diseases, etc.

BioThings SDK

SDK represents "Software Development Kit". BioThings SDK provides a Python-based toolkit to build high-performance data APIs (or web services) from a single data source or multiple data sources. It has the particular focus on building data APIs for biomedical-related entities, a.k.a "BioThings", though it's not necessarily limited to the biomedical scope. For any given "BioThings" type, BioThings SDK helps developers to aggregate annotations from multiple data sources, and expose them as a clean and high-performance web API.

The BioThings SDK can be roughly divided into two main components: data hub (or just "hub") component and web component. The hub component allows developers to automate the process of monitoring, parsing and uploading your data source to an Elasticsearch backend. From here, the web component, built on the high-concurrency Tornado Web Server , allows you to easily setup a live high-performance API. The API endpoints expose simple-to-use yet powerful query features using Elasticsearch's full-text query capabilities and query language.

BioThings API

We also use "BioThings API" (or BioThings APIs) to refer to an API (or a collection of APIs) built with BioThings SDK. For example, both our popular MyGene.Info and MyVariant.Info APIs are built and maintained using this BioThings SDK.

BioThings Studio

BioThings Studio is a buildin, pre-configured environment used to build and administer a BioThings API. At its core is the Hub, a backend service responsible for maintaining data up-to-date, producing data releases and update API frontends.

Installing BioThings SDK

You can install the latest stable BioThings SDK release with pip from PyPI, like:

pip install biothings

You can install the latest development version of BioThings SDK directly from our github repository like:

pip install git+https://github.com/biothings/biothings.api.git#egg=biothings

Alternatively, you can download the source code, or clone the BioThings SDK repository and run:

python setup.py install

Get started to build a BioThings API

We recommend to follow this tutorial to develop your first BioThings API in our pre-configured BioThings Studio development environment.

Documentation

The latest documentation is available at https://docs.biothings.io.

How to contribute

Please check out this Contribution Guidelines and Code of Conduct document.

Comments

Unable to start either [demo_myvariant.docker, old_myvariant.docker]

Following instructions from http://docs.biothings.io/en/latest/doc/standalone.html#quick-links Both images exhibit the same behavior:

hub not starting
- curl http://localhost:19200/_cat/indices (returns nothing)
inability to ssh from host to container
inability to start hub cli

Host information:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04 LTS
Release:	18.04
Codename:	bionic

$ docker --version
Docker version 17.12.1-ce, build 7390fc6

$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 84
Server Version: 17.12.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9b55aab90508bd389d7654c4baf173a981477d55
runc version: 9f9c96235cc97674e935002fc3d78361b696a69e
init version: v0.13.0 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-22-generic
Operating System: Ubuntu 18.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 24
Total Memory: 118GiB
Name: bmeg-build
ID: JRG3:XIRP:VOMU:Z5IM:CBNN:M2TT:QP6J:TE4G:B6V4:C5KI:RZ6T:S7ZX
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false


# docker run --name old_myvariant -p 19080:80 -p 19200:9200 -p 19022:7022 -p 19090:7080 -d old_myvariant
2171a1bd50736f4074c9c3102282ae4b92a8002335347217d65a5e8681b49c3f
[email protected]:/mnt/walsbr#  curl -v http://localhost:19080/metadata
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 19080 (#0)
> GET /metadata HTTP/1.1
> Host: localhost:19080
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 500 Internal Server Error
< Date: Fri, 13 Jul 2018 20:11:23 GMT
< Content-Type: text/html; charset=UTF-8
< Content-Length: 93
< Connection: keep-alive
< Server: TornadoServer/4.5.1
<
* Connection #0 to host localhost left intact
<html><title>500: Internal Server Error</title><body>500: Internal Server Error</body></html>[email protected]:/mnt/walsbr#


[email protected]:/mnt/walsbr#  curl http://localhost:19200/_cat/indices

[email protected]:/mnt/walsbr#  ssh [email protected] -p 19022
ssh_exchange_identification: read: Connection reset by peer


[email protected]:/mnt/walsbr# docker exec -it old_myvariant bash


Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/biothings/myvariant.info/src/biothings/bin/autohub.py", line 16, in <module>
    biothings.config_for_app(config)
  File "/home/biothings/myvariant.info/src/biothings/__init__.py", line 55, in config_for_app
    check_config(config_mod)
  File "/home/biothings/myvariant.info/src/biothings/__init__.py", line 32, in check_config
    raise ConfigurationError("%s: %s" % (attr,str(getattr(config_mod,attr))))
biothings.ConfigurationError: DATA_PLUGIN_FOLDER: Define path to folder which will contain all 3rd party parsers, dumpers, etc...
(pyenv) [email protected]:~/myvariant.info/src$

bug

opened by bwalsh 16

Fetch >1000 documents with a POST query
Originally from @colleenXu:

To find associations between things, we are mostly doing POST queries to biothings apis.

For POST queries, we can retrieve <=1000 records per input (think of a batch-query of input IDs like below). This allows a batch-query to include up to 1000 inputs.

POST to https://mydisease.info/v1/query?fields=disgenet.xrefs,_id&size=1000 with the body: { "q": "7157,7180,7190", "scopes": "disgenet.genes_related_to_disease.gene_id" }

My understanding is that there's only 1 way to change this situation:

CANNOT DO fetch_all, since that only works for GET queries (and just using GET queries isn't a viable solution because not being able to batch-query can slow down multi-hop BTE queries quite a bit).

CAN DO: The only way to get >1000 records per input is to adjust the biothings api settings - which would likely involve lowering the batch-query limit (ex: 10000 records per input and 100 IDs per batch). This can perhaps be done on a per-api basis (like specific pending apis???)

Noting that this has been a discussion topic for a while. And for now, we've been keeping okay at keeping things at <= 1000 records per input, knowing that we are not getting the complete response. This is because it is difficult handling a node attached to lots of other entities...

However, this is known to be more of an issue for APIs that keep many separate records for the same basic association X-related_to-Y. This happens with semmeddb (at least 1 record per publication-association) and some multiomics apis. These are all on the pending api hub.

enhancement
opened by erikyao 12
Automated basic biothings web functionality test for data applications

Now that we have automated tests to run after each app (mygene, myvariant, ...) build under development, explore the possibility to run an automated basic biothings functionality test to ensure in addition to the customizations working, the basic features are also not affected.

opened by namespacestd0 9
FTPDumper does not clean up old downloaded files when ARCHIVE is set to False

When ARCHIVE is set to False, the downloaded files will be saved to the same folder (Data_Archive_root/latest). Looks like FTPDumper does not clean up the old downloads.

Here is an example from mychem hub: PubChemDumper.
bug

opened by newgene 8
remove remaining boto dependency (using boto3 instead)

We still have a few remaining places using boto to access AWS s3 buckets:

https://github.com/biothings/biothings.api/blob/621887f04aae13c3a775aea9aa7daacb92ae7ef0/biothings/utils/aws.py#L6

and

https://github.com/biothings/biothings.api/blob/621887f04aae13c3a775aea9aa7daacb92ae7ef0/biothings/hub/dataexport/ids.py#L4

Most of other AWS related code has been migrated to use boto3. Let's remove the boto dependency completely.
enhancement

opened by newgene 8
display optional "description" in API metadata

If a user wants to know what a given API is for (e.g., "repodb"), the best options now seem to be to search the API name and/or look at example records. We should also allow an optional "description" to be provided in the manifest.json metadata, which would optionally provide some human-readable description directly on the API page.

EDIT: Sorry, just realized I should probably have created this issue in https://github.com/biothings/pending.api/issues.. Feel free to recreate it over there if helpful.
enhancement

opened by andrewsu 7
Replace boto calls to use boto3

This partially addresses Issue #133

All uses except for one usage has been replaced, and tested.

The explanation for the one remaining usage will be documented in comments under Issue #133

opened by zcqian 7

`SnapshotTaskEnv` cannot create `ESIndexer` instances

When I create a new snapshot, the corresponding SnapshotTaskEnv instance cannot initiate due to the failure in the creation of the ESIndexer instance.

Error messages are like:

Aug 16 17:37:09 su06 python[57443]: HTTPServerRequest(protocol='http', host='localhost:19080', method='PUT', uri='/snapshot', version='HTTP/1.1', remote_ip='172.29.80.35')
Aug 16 17:37:09 su06 python[57443]: Traceback (most recent call last):
Aug 16 17:37:09 su06 python[57443]:   File "/opt/home/pending/venv/lib/python3.6/site-packages/tornado/web.py", line 1704, in _execute
Aug 16 17:37:09 su06 python[57443]:     result = await result
Aug 16 17:37:09 su06 python[57443]:   File "<string>", line 69, in put
Aug 16 17:37:09 su06 python[57443]:   File "/opt/home/pending/venv/src/biothings/biothings/hub/dataindex/snapshooter.py", line 521, in snapshot
Aug 16 17:37:09 su06 python[57443]:     return env_for_build.snapshot(index, snapshot=snapshot, steps=steps)
Aug 16 17:37:09 su06 python[57443]:   File "/opt/home/pending/venv/src/biothings/biothings/hub/dataindex/snapshooter.py", line 314, in snapshot
Aug 16 17:37:09 su06 python[57443]:     task_env = SnapshotTaskEnv(self, index, snapshot)
Aug 16 17:37:09 su06 python[57443]:   File "/opt/home/pending/venv/src/biothings/biothings/hub/dataindex/snapshooter.py", line 244, in __init__
Aug 16 17:37:09 su06 python[57443]:     doc_type=env.build_doc['index'][index]['doc_type'],
Aug 16 17:37:09 su06 python[57443]: KeyError: 'doc_type'
Aug 16 17:37:09 su06 python[57443]: ERROR:tornado.access:500 PUT /snapshot (172.29.80.35) 37.72ms

The root causes are:

src_build does not hold the doc_type values anymore
The ESIndexer class definition is out-of-date with ES7

A sample env.build_doc['index'][index] entry is:

        "index" : {
		"idisk_20210812_sicwkhq0" : {
			"host" : "su03:9200",
			"environment" : "su03",
			"created_at" : ISODate("2021-08-16T23:17:07.550Z"),
			"count" : 919
		}
	},

opened by erikyao 6

`GitDumper` should also checkout the `main` branches
When adding a new plugin to pending.biothings.io, the URL to the GitHub repo will be passed to http://localhost:19080/dataplugin/register_url,

which in the end calls the AssistantManager.register_url() method (assistant.py#L699).

The AssistantManager instance looks like to add a message (including the URL to register) to its corresponding MongoDB collection, and finally a GitDumper instance will receive the URL and then check the repo out. By default, GitDumper will only check master branches, but recently GitHub has changed its default branch name from master to main. Therefore our GitDumper cannot checkout the latest GitHub repo-based plugins.

The root cause in the code seems to be dumper.py#L1072:

DEFAULT_BRANCH = "master"
bug enhancement
opened by erikyao 6
Implement full release installation without downtime
Currently from the biothings hub and studio, we can install incremental releases directly with no downtime (applying diffs on the production index). The relevant code is here:

https://github.com/biothings/biothings.api/blob/1b96f0aded05873d642134c0c38b15fa982e3b6d/biothings/hub/standalone/init.py#L68

In the case of deploying a full release, we currently have two options:

delete the old index and then install the new index (restoring snapshots) This causes downtime, but it's brief for small data indices.

we perform a manual index restoration to a different index name and then switch the alias when it's done. This has no downtime, should be preferred.

We should implement these manual steps from the #2 option as a feature in biothings hub/studio.
enhancement
opened by newgene 6
Can manifest-based data plugins and regular data sources stay in the same folder?
Right now, manifest-based data plugins stay in a separate folder like "plugins", and regular dumper/uploader based data sources stay in hub/dataload/sources folder.

Can we allow them in the same folder? It may just work already, let's verify and try to make it work if not.

plugins folder example: https://github.com/biothings/mydisease.info/tree/master/src/plugins

regular data sources example: https://github.com/newgene/biothings_docker/tree/main/tests/hubapi/demohub/biothing_studio/hub/dataload/sources

enhancement
opened by newgene 5
Create an Elasticsearch reindex helper function
This should be a standalone helper function (e.g. can be under utils/es_reindex.py) used only from the Python/iPython console manually when needed. It helps to reindex an existing index by transferring both the settings, mappings and docs.

Elasticsearch's reindex API should be used, however mappings and settings from the old index should be used to create a new empty target index first. Then reindex API can be called to transfer all docs to the new index. Optionally, the alias should be switched over to the new index too. This is useful when we need to migrate existing indices created from the older ES version to the current ES version.

def reindex(src_index, target_index=None, settings=None, mappings=None, alias=None, delete_src=False):

target_index: use <src_index_name>_reindexed as default if None

settings: if provided as a dict, update the settings with the provided dictionary. Otherwise, keep the same from the src_index

mapping: if provided as a dict, update the settings with the provided mappings. Otherwise, keep the same from the src_index

alias: if True, switch the alias from src_index to target_index. If src_index has no alias, apply the <src_index_name> as the alias; if a string value, apply it as the alias instead

delete_src: If True, delete the src_index after everything is done

And after reindex, please also do a refresh & flush and then double-check the doc counts to make sure they are equal.
enhancement
opened by newgene 0
Fix Aggregation Web formatter
Histogram aggregations do not contain the fields aggregations.<term>.doc_count_error_upper_bound, aggregations.<term>.sum_other_doc_count

In our es formatter, it assumes that these two values exist. https://github.com/biothings/biothings.api/blob/master/biothings/web/query/formatter.py#L426-L427

When making a custom aggregation in the esquerybuilder, I have to override these two values.

res[facet]['other'] = res[facet].pop('sum_other_doc_count', 0) res[facet]['missing'] = res[facet].pop('doc_count_error_upper_bound', 0)

We should have this as a quick fix so when other users make custom aggregations they wont have to override the transform_aggs method.
opened by jal347 0
DockerContainerDumper class

This can be a new type of Dumper class, which triggers a docker container (typically runs on a different server) to run and generate the output file, and then stop the container. The dumper class will then get the processed file(s) and send it to the Uploader as normally.

Typically, this processed file can be a NDJSON file (one JSON per line), so the uploader class can be quite simple and generic.

The typical use case is some complex workflow with heavy dependencies, so we can isolate them in a docker container.
enhancement

opened by newgene 0
create biothings.hub API document

Our current documentation site at https://docs.biothings.io/ does not contain API document from biothings.hub module. It was due to some errors in the past, let's re-evaluate to see if we can generate it automatically now.
enhancement

opened by newgene 0
Evaluate and upgrade Elasticsearch v8.x client

for both elasticsearch-py and elasticsearch-dsl packages, their ES8 support is complete. We should test and upgrade.

All of our hubs are now using ES8, but we should target the support of both ES7 and ES8 if possible.
enhancement

opened by newgene 0

Releases(v0.11.1)

v0.11.1(Oct 4, 2022)
This is a bug-fix release with these CHANGES (see also CHANGES.txt):

v0.11.1 (2022/10/03)

Hub improvements:

use pickle protocol 4 as the pickle.dump default

Hub bug fixes:

Fixed a JSON serialization error during incremental release https://github.com/newgene/biothings.api/pull/65

Resolved a hub error when installing a full release https://github.com/biothings/biothings.api/issues/257

Fixed a quick_index error when a data source has multiple uploaders https://github.com/newgene/biothings.api/pull/66

Source code(tar.gz)
Source code(zip)
v0.11.0(Sep 14, 2022)

A major release with many web and hub component enhancement and bug fixes. See CHANGES.txt for details.
Source code(tar.gz)
Source code(zip)
v0.8.5(Jan 8, 2021)

Source code(tar.gz)
Source code(zip)

Owner

BioThings

High Performance Data APIs in Biology

GitHub Repository https://biothings.io

Discord bot for playing blindfold chess.

Albin Discord bot for playing blindfold chess written in Python. Albin takes the moves from chat and pushes them on the board without showing it. TODO

8 Oct 14, 2022

Snipe fair coin launches. Contact @dannsniper on telegram for whitelist

Pancakeswap-sniper Pancakeswap Sniper bot Full version of Pancakeswap sniping bot used to snipe during fair coin launches. With advanced options and a

36 Nov 01, 2021

Discord bot to monitor collection of mods on the Steam Workshop and notify on update to selected discord server via Nextcordbot API.

Steam-Workshop-Monitor Discord bot to monitor collection of mods on the Steam Workshop and notify on update to selected Discord channel via Nextcordbo

7 Nov 03, 2022

Trellox Tool is written in Python3 and designed to pull and list Trello boards.

TrelloX Trellox Tool is written in Python3 and designed to list and pull Trello boards. It can be used by penetration testers/bug bounty hunters to de

1 Dec 05, 2021

Super Fast Telegram UserBot Made With Python.

Description Super Fast Telegram UserBot Made With Python. ＬＯＧＯ Made With Support of All Userbots Dev's Dark-Venom is a Light-Weight Userbot. It's unde

2 Sep 14, 2021

See trending stock tickers on Reddit and check Stock perfomance

1.5k Jan 06, 2023

This is a script to export logs from AWS CloudTrail to a local file.

cloudtrail-export-logs This is a script to export logs from AWS CloudTrail to a local file. Getting Started Prerequisites python 3 boto3 pip Installin

2 Jan 02, 2022

"Nesse projeto criei uma automação para abrir as tarefas no Jira em massa pegando de uma determinada fila do Zendesk."

automacao-Zendesk "Nesse projeto criei uma automação para abrir as tarefas no Jira em massa pegando de uma determinada fila do Zendesk." en-us "In thi

1 Dec 20, 2021

Emo-Fun is a bot which emojifies the text you send it

About Emo-Fun is a bot which emojifies the text you send it. It is easier to understand by an example Input : Hey this is to show my working!! Output

3 Sep 30, 2022

Telegram vc userbot

Telegram Vc Userbot Available Commands /ping :- To check whether userbot is up or not /joinvc :- To join vc /leavevc :- To leave vc /join_group :- To

7 Nov 18, 2022

Python wrapper for Revolt API

defectio is a direct implementation of the entire Revolt API and provides a way to authenticate and start communicating with Revolt servers. Similar interface to discord.py

26 Sep 18, 2022

Autodrive is designed to make it as easy as possible to interact with the Google Drive and Sheets APIs via Python

Autodrive Autodrive is designed to make it as easy as possible to interact with the Google Drive and Sheets APIs via Python. It is especially designed

1 Oct 02, 2021

Reddit bot for r/khiphop

khiphop-bot Description This project is a collection of scripts that better the state of the r/khiphop subreddit, which represents Korean Hip-Hop and

1 Dec 21, 2021

An information scroller Twitter trends, news, weather for raspberry pi and Pimoroni Unicorn Hat Mini and Scroll Phat HD.

uticker An information scroller Twitter trends, news, weather for raspberry pi and Pimoroni Unicorn Hat Mini and Scroll Phat HD. Features include: Twi

5 Oct 31, 2022

Bender: A Markov Babbler Slack Bot

See the Digital Ocean tutorial for instructions on how to get the basic bot structure in place. Once you have that, set the gunicorn command to run as

1 Dec 04, 2021

Telegram bot to stream videos in telegram voicechat for both groups and channels. Supports live strams, YouTube videos and telegram media.

Telegram VCVideoPlayBot An Telegram Bot By @ZauteKm To Stream Videos in Telegram Voice Chat. NOTE: Make sure you have started a VoiceChat in your Grou

20 Oct 21, 2022

BioThings API framework - Making high-performance API for biological annotation data

Related tags

Overview

BioThings SDK

Quick Summary

Introduction

What's BioThings?

BioThings SDK

BioThings API

BioThings Studio

Installing BioThings SDK

Get started to build a BioThings API

Documentation

How to contribute

Comments

Releases(v0.11.1)

v0.11.1(Oct 4, 2022)

v0.11.0(Sep 14, 2022)

v0.8.5(Jan 8, 2021)

Owner

BioThings

Discord bot for playing blindfold chess.

Snipe fair coin launches. Contact @dannsniper on telegram for whitelist

Discord bot to monitor collection of mods on the Steam Workshop and notify on update to selected discord server via Nextcordbot API.

Trellox Tool is written in Python3 and designed to pull and list Trello boards.

Super Fast Telegram UserBot Made With Python.

See trending stock tickers on Reddit and check Stock perfomance

This is a script to export logs from AWS CloudTrail to a local file.

"Nesse projeto criei uma automação para abrir as tarefas no Jira em massa pegando de uma determinada fila do Zendesk."

Emo-Fun is a bot which emojifies the text you send it

Telegram vc userbot

Python wrapper for Revolt API

Autodrive is designed to make it as easy as possible to interact with the Google Drive and Sheets APIs via Python

Reddit bot for r/khiphop

An information scroller Twitter trends, news, weather for raspberry pi and Pimoroni Unicorn Hat Mini and Scroll Phat HD.

Bender: A Markov Babbler Slack Bot

Telegram bot to stream videos in telegram voicechat for both groups and channels. Supports live strams, YouTube videos and telegram media.

MSE5050/7050 Materials Informatics course at the University of Utah

Retrieves GitHub Stats via `git_api` and flask.

Fortnite Dumper for anyone's Save the World profiles.

A Python Client to View F1TV Content the right way