open source tools to generate mypy stubs from protobufs

Overview

mypy-protobuf: Generate mypy stub files from protobuf specs

CI pypi license

We just released a new major release mypy-protobuf 2. on 02/02/2021! It includes some backward incompatible changes. See Changelog for recent changes.

Requirements

mypy >= v0.800 protoc 3.14.0 or greater python-protobuf >= 3.14.0 python >= 3.6 - for running mypy-protobuf plugin. Generated stubs will compatible back to python2.

Other configurations may work, but are not supported in testing currently. We would be open to expanding this list if a need arises - file an issue on the issue tracker.

Installation

The plugin can be installed with

pip install mypy-protobuf

To install unreleased

REV=master  # or whichever unreleased git rev you'd like
pip install git+https://github.com/dropbox/mypy-protobuf.git@$REV

# Prior to directory structure flattening, you may need
pip install git+https://github.com/dropbox/mypy-protobuf.git@$REV#subdirectory=python

Implementation

The implementation of the plugin is in mypy_protobuf/main.py, which installs to a posix executable protoc-gen-mypy. On windows you will have to use protoc_gen_mypy.bat for the executable.

On posix, ensure that the protoc-gen-mypy script installed onto your $PATH. Then run.

protoc --python_out=output/location --mypy_out=output/location

Alternately, you can explicitly provide the path:

protoc --plugin=protoc-gen-mypy=path/to/protoc-gen-mypy --python_out=output/location --mypy_out=output/location

On windows, provide the bat file:

protoc --plugin=protoc-gen-mypy=path/to/protoc_gen_mypy.bat --python_out=output/location --mypy_out=output/location

Features

See Changelog for full listing

Types enum int values more strongly

Enum int values produce stubs which wrap the int values in NewType

enum MyEnum {
  FOO = 0;
  BAR = 1;
}

Will yield an enum type wrapper whose methods type to MyEnum.V rather than int. This allows mypy to catch bugs where the wrong enum value is being used.

mypy-protobuf autogenerates an instance of the EnumTypeWrapper as follows.

class _MyEnum(google.protobuf.internal.EnumTypeWrapper[MyEnum.V], builtins.type):
    DESCRIPTOR: google___protobuf___descriptor___EnumDescriptor = ...
    FOO = MyEnum.V(0)
    BAR = MyEnum.V(1)
class MyEnum(metaclass=_OuterEnum):
    V = typing___NewType('V', builtins.int)
FOO = MyEnum.V(0)
BAR = MyEnum.V(1)

Calling code may be typed as follows. Note that the type of x must be quoted until upstream protobuf supports V

def f(x: 'MyEnum.V'):
    print(x)

f(MyEnum.Value("FOO"))

Supports generating type wrappers for fields and maps

M.proto

message M {
  uint32 user_id = 1 [(mypy_protobuf.casttype)="mymod.UserId"
  map<uint32, string> email_by_uid = 2 [
    (mypy_protobuf.keytype)="path/to/mymod.UserId",
    (mypy_protobuf.valuetype)="path/to/mymod.Email"
  ];
}

mymod.py

UserId = NewType("UserId", int)
Email = NewType("Email", Text)

py_generic_services

If py_generic_services is set in your proto file, then mypy-protobuf will generate service stubs. If you want GRPC stubs instead - use the GRPC instructions.

readable_stubs

If readable_stubs is set, mypy-protobuf will generate easier-to-read stubs. The downside to this approach - is that it's possible to generate stubs which do not pass mypy - particularly in the case of name collisions. mypy-protobuf defaults to generating stubs with fully qualified imports and mangled global-level identifiers to defend against name collisions between global identifiers and field names.

If you're ok with this risk, try it out!

protoc --python_out=output/location --mypy_out=readable_stubs:output/location

relax_strict_optional_primitives

If you are using proto3, then primitives cannot be represented as NULL on the wire - only as their zero value. By default mypy-protobuf types message constructors to have non-nullable primitives (eg int instead of Optional[int]). python-protobuf itself will internally convert None -> zero value, and if you intentionally want to use this behavior, set this! We recommend avoiding this, but it may be helpful when migrating existing proto2 code.

protoc --python_out=output/location --mypy_out=relax_strict_optional_primitives:output/location

Output suppression

To suppress output, you can run

protoc --python_out=output/location --mypy_out=quiet:output/location

GRPC

This plugin provides stubs generation for grpcio generated code.

protoc \
    --python_out=output/location \
    --mypy_out=output/location \
    --grpc_out=output/location \
    --mypy_grpc_out=output/location

Note that generated code for grpc will work only together with code for python and locations should be the same. If you need stubs for grpc internal code we suggest using this package https://github.com/shabbyrobe/grpc-stubs

Contributing

Contributions to the implementation are welcome. Please run tests using ./run_test.sh. Ensure code is formatted using black.

pip3 install black
black mypy_protobuf/main.py test/

Contributors

Dropboxers

Others

Licence etc.

  1. License: Apache 2.0.
  2. Copyright attribution: Copyright (c) 2017 Dropbox, Inc.
  3. External contributions to the project should be subject to Dropbox's Contributor License Agreement (CLA): https://opensource.dropbox.com/cla/
Owner
Dropbox
Dropbox
Mypy plugin and stubs for SQLAlchemy

Pythonista Stubs Stubs for the Pythonista iOS API. This allows for better error detection and IDE / editor autocomplete. Installation and Usage pip in

Dropbox 521 Dec 29, 2022
❄️ A flake8 plugin to help you write better list/set/dict comprehensions.

flake8-comprehensions A flake8 plugin that helps you write better list/set/dict comprehensions. Requirements Python 3.6 to 3.9 supported. Installation

Adam Johnson 398 Dec 23, 2022
Performant type-checking for python.

Pyre is a performant type checker for Python compliant with PEP 484. Pyre can analyze codebases with millions of lines of code incrementally – providi

Facebook 6.2k Jan 04, 2023
Naming Convention checker for Python

PEP 8 Naming Conventions Check your code against PEP 8 naming conventions. This module provides a plugin for flake8, the Python code checker. (It repl

Python Code Quality Authority 411 Dec 23, 2022
🦆 Better duck-typing with mypy-compatible extensions to Protocol

🦆 Quacks If it walks like a duck and it quacks like a duck, then it must be a duck Thanks to PEP544, Python now has protocols: a way to define duck t

Arie Bovenberg 9 Nov 14, 2022
Static Typing for Python

Python static typing home. Contains the source for typing_extensions and the documentation. Also hosts a user help forum.

Python 1.3k Jan 06, 2023
Pylint plugin for improving code analysis for when using Django

pylint-django About pylint-django is a Pylint plugin for improving code analysis when analysing code using Django. It is also used by the Prospector t

Python Code Quality Authority 544 Jan 06, 2023
A simple program which checks Python source files for errors

Pyflakes A simple program which checks Python source files for errors. Pyflakes analyzes programs and detects various errors. It works by parsing the

Python Code Quality Authority 1.2k Dec 30, 2022
Type annotations builder for boto3 compatible with VSCode, PyCharm, Emacs, Sublime Text, pyright and mypy.

mypy_boto3_builder Type annotations builder for boto3-stubs project. Compatible with VSCode, PyCharm, Emacs, Sublime Text, mypy, pyright and other too

Vlad Emelianov 2 Dec 05, 2022
Design by contract for Python. Write bug-free code. Add a few decorators, get static analysis and tests for free.

A Python library for design by contract (DbC) and checking values, exceptions, and side-effects. In a nutshell, deal empowers you to write bug-free co

Life4 473 Dec 28, 2022
Flake8 extension to provide force-check option

flake8-force Flake8 extension to provide force-check option. When this option is enabled, flake8 performs all checks even if the target file cannot be

Kenichi Maehashi 9 Oct 29, 2022
A plugin for Flake8 that provides specializations for type hinting stub files

flake8-pyi A plugin for Flake8 that provides specializations for type hinting stub files, especially interesting for linting typeshed. Functionality A

Łukasz Langa 58 Jan 04, 2023
It's not just a linter that annoys you!

README for Pylint - https://pylint.pycqa.org/ Professional support for pylint is available as part of the Tidelift Subscription. Tidelift gives softwa

Python Code Quality Authority 4.4k Jan 04, 2023
Stubs with type annotations for ordered-set Python library

ordered-set-stubs - stubs with type annotations for ordered-set Python library Archived - now type annotations are the part of the ordered-set library

Roman Inflianskas 2 Feb 06, 2020
The strictest and most opinionated python linter ever!

wemake-python-styleguide Welcome to the strictest and most opinionated python linter ever. wemake-python-styleguide is actually a flake8 plugin with s

wemake.services 2.1k Jan 01, 2023
A Python Parser

parso - A Python Parser Parso is a Python parser that supports error recovery and round-trip parsing for different Python versions (in multiple Python

Dave Halter 520 Dec 26, 2022
Easy saving and switching between multiple KDE configurations.

Konfsave Konfsave is a config manager. That is, it allows you to save, back up, and easily switch between different (per-user) system configurations.

42 Sep 25, 2022
Typed interface stubs for Pythonista iOS

Pythonista Stubs Stubs for the Pythonista iOS API. This allows for better error detection and IDE / editor autocomplete. Installation and Usage pip in

Harold Martin 12 Jul 14, 2020
A python documentation linter which checks that the docstring description matches the definition.

Darglint A functional docstring linter which checks whether a docstring's description matches the actual function/method implementation. Darglint expe

Terrence Reilly 463 Dec 31, 2022
Pyright extension for coc.nvim

coc-pyright Pyright extension for coc.nvim Install :CocInstall coc-pyright Note: Pyright may not work as expected if can't detect project root correct

Heyward Fann 1.1k Jan 02, 2023