Auto-detecting the n+1 queries problem in Python

Overview

nplusone

Latest version Travis-CI Code coverage

nplusone is a library for detecting the n+1 queries problem in Python ORMs, including SQLAlchemy, Peewee, and the Django ORM.

The Problem

Many object-relational mapping (ORM) libraries default to lazy loading for relationships. This pattern can be efficient when related rows are rarely accessed, but quickly becomes inefficient as relationships are accessed more frequently. In these cases, loading related rows eagerly using a JOIN can be vastly more performant. Unfortunately, understanding when to use lazy versus eager loading can be challenging: you might not notice the problem until your app has slowed to a crawl.

nplusone is an ORM profiling tool to help diagnose and improve poor performance caused by inappropriate lazy loading. nplusone monitors applications using Django or SQLAlchemy and sends notifications when potentially expensive lazy loads are emitted. It can identify the offending relationship attribute and specific lines of code behind the problem, and recommend fixes for better performance.

nplusone also detects inappropriate eager loading for Flask-SQLAlchemy and the Django ORM, emitting a warning when related data are eagerly loaded but never accessed within the current request.

Installation

pip install -U nplusone

nplusone supports Python >= 2.7 or >= 3.3.

Usage

Note: nplusone should only be used for development and should not be deployed to production environments.

Django

Note: nplusone supports Django >= 1.8.

Add nplusone to INSTALLED_APPS:

INSTALLED_APPS = (
    ...
    'nplusone.ext.django',
)

Add NPlusOneMiddleware:

MIDDLEWARE = (
    'nplusone.ext.django.NPlusOneMiddleware',
    ...
)

Optionally configure logging settings:

NPLUSONE_LOGGER = logging.getLogger('nplusone')
NPLUSONE_LOG_LEVEL = logging.WARN

Configure logging handlers:

LOGGING = {
    'version': 1,
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
        },
    },
    'loggers': {
        'nplusone': {
            'handlers': ['console'],
            'level': 'WARN',
        },
    },
}

When your app loads data lazily, nplusone will emit a log message:

Potential n+1 query detected on `<model>.<field>`

Consider using select_related or prefetch_related in this case.

When your app eagerly loads related data without accessing it, nplusone will log a warning:

Potential unnecessary eager load detected on `<model>.<field>`

Flask-SQLAlchemy

Wrap application with NPlusOne:

from flask import Flask
from nplusone.ext.flask_sqlalchemy import NPlusOne

app = Flask(__name__)
NPlusOne(app)

Optionally configure logging settings:

app = Flask(__name__)
app.config['NPLUSONE_LOGGER'] = logging.getLogger('app.nplusone')
app.config['NPLUSONE_LOG_LEVEL'] = logging.ERROR
NPlusOne(app)

When your app loads data lazily, nplusone will emit a log message:

Potential n+1 query detected on `<model>.<field>`

Consider using subqueryload or joinedload in this case; see SQLAlchemy's guide to relationship loading for complete documentation.

When your app eagerly loads related data without accessing it, nplusone will log a warning:

Potential unnecessary eager load detected on `<model>.<field>`

WSGI

For other frameworks that follow the WSGI specification, wrap your application with NPlusOneMiddleware. You must also import the relevant nplusone extension for your ORM:

import bottle
from nplusone.ext.wsgi import NPlusOneMiddleware
import nplusone.ext.sqlalchemy

app = NPlusOneMiddleware(bottle.app())

Generic

The integrations above are coupled to the request-response cycle. To use nplusone outside the context of an HTTP request, use the Profiler context manager: You must also import the relevant nplusone extension for your ORM:

from nplusone.core import profiler
import nplusone.ext.sqlalchemy

with profiler.Profiler():
    ...

Customizing notifications

By default, nplusone logs all potentially unnecessary queries using a logger named "nplusone". When the NPLUSONE_RAISE configuration option is set, nplusone will also raise an NPlusOneError. This can be used to force all automated tests involving unnecessary queries to fail.

# Django config
NPLUSONE_RAISE = True

# Flask config
app.config['NPLUSONE_RAISE'] = True

The exception type can also be specified, if desired, using the NPLUSONE_ERROR option.

Ignoring notifications

To ignore notifications from nplusone globally, configure the whitelist using the NPLUSONE_WHITELIST option:

# Django config
NPLUSONE_WHITELIST = [
    {'label': 'n_plus_one', 'model': 'myapp.MyModel'}
]

# Flask-SQLAlchemy config
app.config['NPLUSONE_WHITELIST'] = [
    {'label': 'unused_eager_load', 'model': 'MyModel', 'field': 'my_field'}
]

You can whitelist models by exact name or by fnmatch patterns:

# Django config
NPLUSONE_WHITELIST = [
    {'model': 'myapp.*'}
]

To suppress notifications locally, use the ignore context manager:

from nplusone.core import signals

with signals.ignore(signals.lazy_load):
    # lazy-load rows
    # ...

License

MIT licensed. See the bundled LICENSE file for more details.

Owner
Joshua Carp
Joshua Carp
A gdb-like Python3 Debugger in the Trepan family

Abstract Features More Exact location information Debugging Python bytecode (no source available) Source-code Syntax Colorization Command Completion T

R. Bernstein 126 Nov 24, 2022
A powerful set of Python debugging tools, based on PySnooper

snoop snoop is a powerful set of Python debugging tools. It's primarily meant to be a more featureful and refined version of PySnooper. It also includ

Alex Hall 874 Jan 08, 2023
An improbable web debugger through WebSockets

wdb - Web Debugger Description wdb is a full featured web debugger based on a client-server architecture. The wdb server which is responsible of manag

Kozea 1.6k Dec 09, 2022
Graphical Python debugger which lets you easily view the values of all evaluated expressions

birdseye birdseye is a Python debugger which records the values of expressions in a function call and lets you easily view them after the function exi

Alex Hall 1.5k Dec 24, 2022
🔥 Pyflame: A Ptracing Profiler For Python. This project is deprecated and not maintained.

Pyflame: A Ptracing Profiler For Python (This project is deprecated and not maintained.) Pyflame is a high performance profiling tool that generates f

Uber Archive 3k Jan 07, 2023
An x86 old-debug-like program.

An x86 old-debug-like program.

Pablo Niklas 1 Jan 10, 2022
Trace any Python program, anywhere!

lptrace lptrace is strace for Python programs. It lets you see in real-time what functions a Python program is running. It's particularly useful to de

Karim Hamidou 687 Nov 20, 2022
Monitor Memory usage of Python code

Memory Profiler This is a python module for monitoring memory consumption of a process as well as line-by-line analysis of memory consumption for pyth

Fabian Pedregosa 80 Nov 18, 2022
Silky smooth profiling for Django

Silk Silk is a live profiling and inspection tool for the Django framework. Silk intercepts and stores HTTP requests and database queries before prese

Jazzband 3.7k Jan 01, 2023
Django package to log request values such as device, IP address, user CPU time, system CPU time, No of queries, SQL time, no of cache calls, missing, setting data cache calls for a particular URL with a basic UI.

django-web-profiler's documentation: Introduction: django-web-profiler is a django profiling tool which logs, stores debug toolbar statistics and also

MicroPyramid 77 Oct 29, 2022
printstack is a Python package that adds stack trace links to the builtin print function, so that editors such as PyCharm can link you to the source of the print call.

printstack is a Python package that adds stack trace links to the builtin print function, so that editors such as PyCharm can link to the source of the print call.

101 Aug 26, 2022
Dahua Console, access internal debug console and/or other researched functions in Dahua devices.

Dahua Console, access internal debug console and/or other researched functions in Dahua devices.

bashis 156 Dec 28, 2022
A drop-in replacement for Django's runserver.

About A drop in replacement for Django's built-in runserver command. Features include: An extendable interface for handling things such as real-time l

David Cramer 1.3k Dec 15, 2022
一个小脚本,用于trace so中native函数的调用。

trace_natives 一个IDA小脚本,获取SO代码段中所有函数的偏移地址,再使用frida-trace 批量trace so函数的调用。 使用方法 1.将traceNatives.py丢进IDA plugins目录中 2.IDA中,Edit-Plugins-traceNatives IDA输

296 Dec 28, 2022
Trace all method entries and exits, the exit also prints the return value, if it is of basic type

Trace all method entries and exits, the exit also prints the return value, if it is of basic type. The apk must have set the android:debuggable="true" flag.

Kurt Nistelberger 7 Aug 10, 2022
Sentry is cross-platform application monitoring, with a focus on error reporting.

Users and logs provide clues. Sentry provides answers. What's Sentry? Sentry is a service that helps you monitor and fix crashes in realtime. The serv

Sentry 32.9k Dec 31, 2022
pdb++, a drop-in replacement for pdb (the Python debugger)

pdb++, a drop-in replacement for pdb What is it? This module is an extension of the pdb module of the standard library. It is meant to be fully compat

1k Jan 02, 2023
Winpdb Reborn - A GPL Python Debugger, reborn from the unmaintained Winpdb

Note from Philippe Fremy The port of winpdb-reborn to Python 3 / WxPython 4 is unfortunately not working very well. So Winpdb for Python 3 does not re

Philippe F 84 Dec 22, 2022
Hypothesis debugging with vscode

Hypothesis debugging with vscode

Oliver Mannion 0 Feb 09, 2022
Inject code into running Python processes

pyrasite Tools for injecting arbitrary code into running Python processes. homepage: http://pyrasite.com documentation: http://pyrasite.rtfd.org downl

Luke Macken 2.7k Jan 08, 2023