A pytest plugin, that enables you to test your code that relies on a running PostgreSQL Database

Overview

https://raw.githubusercontent.com/ClearcodeHQ/pytest-postgresql/master/logo.png

pytest-postgresql

Latest PyPI version Wheel Status Supported Python Versions License

What is this?

This is a pytest plugin, that enables you to test your code that relies on a running PostgreSQL Database. It allows you to specify fixtures for PostgreSQL process and client.

How to use

Warning

Tested on PostgreSQL versions >= 10. See tests for more details.

Install with:

pip install pytest-postgresql

You will also need to install psycopg. See its installation instructions.

Plugin contains three fixtures:

  • postgresql - it's a client fixture that has functional scope. After each test it ends all leftover connections, and drops test database from PostgreSQL ensuring repeatability. This fixture returns already connected psycopg connection.
  • postgresql_proc - session scoped fixture, that starts PostgreSQL instance at it's first use and stops at the end of the tests.
  • postgresql_noproc - a noprocess fixture, that's connecting to already running postgresql instance. For example on dockerized test environments, or CI providing postgresql services

Simply include one of these fixtures into your tests fixture list.

You can also create additional postgresql client and process fixtures if you'd need to:

from pytest_postgresql import factories

postgresql_my_proc = factories.postgresql_proc(
    port=None, unixsocketdir='/var/run')
postgresql_my = factories.postgresql('postgresql_my_proc')

Note

Each PostgreSQL process fixture can be configured in a different way than the others through the fixture factory arguments.

Sample test

def test_example_postgres(postgresql):
    """Check main postgresql fixture."""
    cur = postgresql.cursor()
    cur.execute("CREATE TABLE test (id serial PRIMARY KEY, num integer, data varchar);")
    postgresql.commit()
    cur.close()

If you want the database fixture to be automatically populated with your schema there are two ways:

  1. client fixture specific
  2. process fixture specific

Both are accepting same set of possible loaders:

  • sql file path
  • loading function import path (string)
  • actual loading function

That function will receive host, port, user, dbname and password kwargs and will have to perform connection to the database inside. However, you'll be able to run SQL files or even trigger programmatically database migrations you have.

Client specific loads the database each test

postgresql_my_with_schema = factories.postgresql(
    'postgresql_my_proc',
    load=["schemafile.sql", "otherschema.sql", "import.path.to.function", "import.path.to:otherfunction", load_this]
)

Warning

This way, the database will still be dropped each time.

The process fixture performs the load once per test session, and loads the data into the template database. Client fixture then creates test database out of the template database each test, which significantly speeds up the tests.

postgresql_my_proc = factories.postgresql_proc(
    load=["schemafile.sql", "otherschema.sql", "import.path.to.function", "import.path.to:otherfunction", load_this]
)
pytest --postgresql-populate-template=path.to.loading_function --postgresql-populate-template=path.to.other:loading_function --postgresql-populate-template=path/to/file.sql

The loading_function from example will receive , and have to commit that. Connecting to already existing postgresql database --------------------------------------------------

Some projects are using already running postgresql servers (ie on docker instances). In order to connect to them, one would be using the postgresql_noproc fixture.

postgresql_external = factories.postgresql('postgresql_noproc')

By default the postgresql_noproc fixture would connect to postgresql instance using 5432 port. Standard configuration options apply to it.

These are the configuration options that are working on all levels with the postgresql_noproc fixture:

Configuration

You can define your settings in three ways, it's fixture factory argument, command line option and pytest.ini configuration option. You can pick which you prefer, but remember that these settings are handled in the following order:

  • Fixture factory argument
  • Command line option
  • Configuration option in your pytest.ini file
Configuration options
PostgreSQL option Fixture factory argument Command line option pytest.ini option Noop process fixture Default
Path to executable executable --postgresql-exec postgresql_exec
/usr/lib/postgresql/13/bin/pg_ctl
host host --postgresql-host postgresql_host yes 127.0.0.1
port port --postgresql-port postgresql_port yes (5432) random
postgresql user user --postgresql-user postgresql_user yes postgres
password password --postgresql-password postgresql_password yes  
Starting parameters (extra pg_ctl arguments) startparams --postgresql-startparams postgresql_startparams
-w
Postgres exe extra arguments (passed via pg_ctl's -o argument) postgres_options --postgresql-postgres-options postgresql_postgres_options
 
Log filename's prefix logsprefix --postgresql-logsprefix postgresql_logsprefix
 
Location for unixsockets unixsocket --postgresql-unixsocketdir postgresql_unixsocketdir
$TMPDIR
Database name dbname --postgresql-dbname postgresql_dbname yes, however with xdist an index is being added to name, resulting in test0, test1 for each worker. test
Default Schema either in sql files or import path to function that will load it (list of values for each) load --postgresql-load postgresql_load yes  
PostgreSQL connection options options --postgresql-options postgresql_options yes  

Example usage:

  • pass it as an argument in your own fixture

    postgresql_proc = factories.postgresql_proc(
        port=8888)
  • use --postgresql-port command line option when you run your tests

    py.test tests --postgresql-port=8888
    
  • specify your port as postgresql_port in your pytest.ini file.

    To do so, put a line like the following under the [pytest] section of your pytest.ini:

    [pytest]
    postgresql_port = 8888

Examples

Populating database for tests

With SQLAlchemy

This example shows how to populate database and create an SQLAlchemy's ORM connection:

Sample below is simplified session fixture from pyramid_fullauth tests:

from sqlalchemy import create_engine
from sqlalchemy.orm import scoped_session, sessionmaker
from sqlalchemy.pool import NullPool
from zope.sqlalchemy import register


@pytest.fixture
def db_session(postgresql):
    """Session for SQLAlchemy."""
    from pyramid_fullauth.models import Base

    connection = f'postgresql+psycopg2://{postgresql.info.user}:@{postgresql.info.host}:{postgresql.info.port}/{postgresql.info.dbname}'

    engine = create_engine(connection, echo=False, poolclass=NullPool)
    pyramid_basemodel.Session = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))
    pyramid_basemodel.bind_engine(
        engine, pyramid_basemodel.Session, should_create=True, should_drop=True)

    yield pyramid_basemodel.Session

    transaction.commit()
    Base.metadata.drop_all(engine)


@pytest.fixture
def user(db_session):
    """Test user fixture."""
    from pyramid_fullauth.models import User
    from tests.tools import DEFAULT_USER

    new_user = User(**DEFAULT_USER)
    db_session.add(new_user)
    transaction.commit()
    return new_user


def test_remove_last_admin(db_session, user):
    """
    Sample test checks internal login, but shows usage in tests with SQLAlchemy
    """
    user = db_session.merge(user)
    user.is_admin = True
    transaction.commit()
    user = db_session.merge(user)

    with pytest.raises(AttributeError):
        user.is_admin = False

Note

See the original code at pyramid_fullauth's conftest file. Depending on your needs, that in between code can fire alembic migrations in case of sqlalchemy stack or any other code

Maintaining database state outside of the fixtures

It is possible and appears it's used in other libraries for tests, to maintain database state with the use of the pytest-postgresql database managing functionality:

For this import DatabaseJanitor and use its init and drop methods:

import pytest
from pytest_postgresql.janitor import DatabaseJanitor

@pytest.fixture
def database(postgresql_proc):
    # variable definition

    janitor = DatabaseJanitor(
        postgresql_proc.user,
        postgresql_proc.host,
        postgresql_proc.port,
        "my_test_database",
        postgresql_proc.version,
        password="secret_password,
    ):
    janitor.init()
    yield psycopg2.connect(
        dbname="my_test_database",
        user=postgresql_proc.user,
        password="secret_password",
        host=postgresql_proc.host,
        port=postgresql_proc.port,
    )
    janitor.drop()

or use it as a context manager:

import pytest
from pytest_postgresql.janitor import DatabaseJanitor

@pytest.fixture
def database(postgresql_proc):
    # variable definition

    with DatabaseJanitor(
        postgresql_proc.user,
        postgresql_proc.host,
        postgresql_proc.port,
        "my_test_database",
        postgresql_proc.version,
        password="secret_password,
    ):
        yield psycopg2.connect(
            dbname="my_test_database",
            user=postgresql_proc.user,
            password="secret_password",
            host=postgresql_proc.host,
            port=postgresql_proc.port,
        )

Note

DatabaseJanitor manages the state of the database, but you'll have to create connection to use in test code yourself.

You can optionally pass in a recognized postgresql ISOLATION_LEVEL for additional control.

Note

See DatabaseJanitor usage in python's warehouse test code https://github.com/pypa/warehouse/blob/5d15bfe/tests/conftest.py#L127

Connecting to Postgresql (in a docker)

To connect to a docker run postgresql and run test on it, use noproc fixtures.

docker run --name some-postgres -e POSTGRES_PASSWORD=mysecretpassword -d postgres

This will start postgresql in a docker container, however using a postgresql installed locally is not much different.

In tests, make sure that all your tests are using postgresql_noproc fixture like that:

postgresql_in_docker = factories.postgresql_noproc()
postresql = factories.postgresql("postgresql_in_docker", db_name="test")


def test_postgres_docker(postresql):
    """Run test."""
    cur = postgresql.cursor()
    cur.execute("CREATE TABLE test (id serial PRIMARY KEY, num integer, data varchar);")
    postgresql.commit()
    cur.close()

And run tests:

pytest --postgresql-host=172.17.0.2 --postgresql-password=mysecretpassword

Using a common database initialisation between tests

If you've got several tests that require common initialisation, you need to define a load and pass it to your custom postgresql process fixture:

import pytest_postgresql.factories
def load_database(**kwargs):
    db_connection: connection = psycopg2.connect(**kwargs)
    with db_connection.cursor() as cur:
        cur.execute("CREATE TABLE stories (id serial PRIMARY KEY, name varchar);")
        cur.execute(
            "INSERT INTO stories (name) VALUES"
            "('Silmarillion'), ('Star Wars'), ('The Expanse'), ('Battlestar Galactica')"
        )
        db_connection.commit()

postgresql_proc = factories.postgresql_proc(
    load=[load_database],
)

postgresql = factories.postgresql(
    "postgresql_proc",
)

You can also define your own database name by passing same dbname value to both factories.

The way this will work is that the process fixture will populate template database, which in turn will be used automatically by client fixture to create a test database from scratch. Fast, clean and no dangling transactions, that could be accidentally rolled back.

Same approach will work with noproces fixture, while connecting to already running postgresql instance whether it'll be on a docker machine or running remotely or locally.

Owner
Clearcode
Software house with a passion for technology. We specialize in building enterprise-grade adtech, martech and analytics platforms.
Clearcode
Let your Python tests travel through time

FreezeGun: Let your Python tests travel through time FreezeGun is a library that allows your Python tests to travel through time by mocking the dateti

Steve Pulec 3.5k Dec 29, 2022
Testing - Instrumenting Sanic framework with Opentelemetry

sanic-otel-splunk Testing - Instrumenting Sanic framework with Opentelemetry Test with python 3.8.10, sanic 20.12.2 Step to instrument pip install -r

Donler 1 Nov 26, 2021
Percy visual testing for Python Selenium

percy-selenium-python Percy visual testing for Python Selenium. Installation npm install @percy/cli: $ npm install --save-dev @percy/cli pip install P

Percy 9 Mar 24, 2022
Aioresponses is a helper for mock/fake web requests in python aiohttp package.

aioresponses Aioresponses is a helper to mock/fake web requests in python aiohttp package. For requests module there are a lot of packages that help u

402 Jan 06, 2023
A Django plugin for pytest.

Welcome to pytest-django! pytest-django allows you to test your Django project/applications with the pytest testing tool. Quick start / tutorial Chang

pytest-dev 1.1k Dec 31, 2022
Python Moonlight (Machine Learning) Practice

PyML Python Moonlight (Machine Learning) Practice Contents Design Documentation Prerequisites Checklist Dev Setup Testing Run Prerequisites Python 3 P

Dockerian Seattle 2 Dec 25, 2022
Turn any OpenAPI2/3 and Postman Collection file into an API server with mocking, transformations and validations.

Prism is a set of packages for API mocking and contract testing with OpenAPI v2 (formerly known as Swagger) and OpenAPI v3.x. Mock Servers: Life-like

Stoplight 3.3k Jan 05, 2023
To automate the generation and validation tests of COSE/CBOR Codes and it's base45/2D Code representations

To automate the generation and validation tests of COSE/CBOR Codes and it's base45/2D Code representations, a lot of data has to be collected to ensure the variance of the tests. This respository was

160 Jul 25, 2022
An Instagram bot that can mass text users, receive and read a text, and store it somewhere with user details.

Instagram Bot 🤖 July 14, 2021 Overview 👍 A multifunctionality automated instagram bot that can mass text users, receive and read a message and store

Abhilash Datta 14 Dec 06, 2022
Enabling easy statistical significance testing for deep neural networks.

deep-significance: Easy and Better Significance Testing for Deep Neural Networks Contents ⁉️ Why 📥 Installation 🔖 Examples Intermezzo: Almost Stocha

Dennis Ulmer 270 Dec 20, 2022
Browser reload with uvicorn

uvicorn-browser This project is inspired by autoreload. Installation pip install uvicorn-browser Usage Run uvicorn-browser --help to see all options.

Marcelo Trylesinski 64 Dec 17, 2022
Hypothesis is a powerful, flexible, and easy to use library for property-based testing.

Hypothesis Hypothesis is a family of testing libraries which let you write tests parametrized by a source of examples. A Hypothesis implementation the

Hypothesis 6.4k Jan 05, 2023
Python tools for penetration testing

pyTools_PT python tools for penetration testing Please don't use these tool for illegal purposes. These tools is meant for penetration testing for leg

Gourab 1 Dec 01, 2021
A folder automation made using Watch-dog, it only works in linux for now but I assume, it will be adaptable to mac and PC as well

folder-automation A folder automation made using Watch-dog, it only works in linux for now but I assume, it will be adaptable to mac and PC as well Th

Parag Jyoti Paul 31 May 28, 2021
Using openpyxl in Python, performed following task

Python-Automation-with-openpyxl Using openpyxl in Python, performed following tasks on an Excel Sheet containing Product Suppliers along with their pr

1 Apr 06, 2022
🏃💨 For when you need to fill out feedback in the last minute.

BMSCE Auto Feedback For when you need to fill out feedback in the last minute. 🏃 💨 Setup Clone the repository Run pip install selenium Set the RATIN

Shaan Subbaiah 10 May 23, 2022
pytest plugin for distributed testing and loop-on-failures testing modes.

xdist: pytest distributed testing plugin The pytest-xdist plugin extends pytest with some unique test execution modes: test run parallelization: if yo

pytest-dev 1.1k Dec 30, 2022
Tools for test driven data-wrangling and data validation.

datatest: Test driven data-wrangling and data validation Datatest helps to speed up and formalize data-wrangling and data validation tasks. It impleme

269 Dec 16, 2022
Spam the buzzer and upgrade automatically - Selenium

CookieClicker Usage: Let's check your chrome navigator version : Consequently, you have to : download the right chromedriver in the follow link : http

Iliam Amara 1 Nov 22, 2021
Android automation project with pytest+appium

Android automation project with pytest+appium

1 Oct 28, 2021