bubblewrap

a wrapper around pytest for assessing flakiness and runtime regressions

a cs implementations practice project

How to Run:

First, install depedencies

# from the bubblewrap root
# first, install depdencies:
$ pip3 install -r requirements.txt`

# then, invoke with:
$ ./bubblewrap path/to/code

e.g.

$ ./bubblewrap .

or, if you're using add bubblewrap to your $PATH and call with just bubblewrap path/to/code, e.g.:

$ PATH="$HOME/path/to/bubblewrap:$PATH"

Demos

This package also has a couple quick demos for testing against:

# from the bubblewrap root
$ ./bubblewrap examples/stable

# from the bubblewrap root
$ ./bubblewrap examples/flake

Usage Notes

As of right now, imports in test modules need to import the full path of the module, e.g.:

# this is alright: 
from example.one.A import apple
# but this is currently supported:
from A import apple

This'll be changed in the future.

Also, bubblewrap doesn't yet support regression analysis

Local Setup

This project uses Python3.9, pip to manage dependencies, and runs in a virtual environment. Install with the following:

# create + activate venv + install dependencies
# see docs:  https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/
$ python3 -m pip install --user virtualenv
$ python3 -m venv env
$ source env/bin/activate
$ pip install -r requirements.txt

# note: to deactivate venv, use
$ deactivate

Code Formatting

This project uses Black for code formatting:

$ python3 -m black .

The pyproject.toml file includes configuration for Black.

And, the project is linted with flake8

$ python3 -m flake8 path/to/code

Tests

Actual unit tests for this project are also run with with pytest. Having installed the pip dependencies (see prior section), run with the following:

# unit tests, ignoring ones in the demo dirs that are designed to flake
$ python3 -m pytest -vv -k 'not flake' .

# test coverage report
$ python3 -m coverage run -m pytest -k 'not flake' .
$ coverage report -m