Protocol Buffers for the Rest of Us

Last update: Jan 04, 2023

Related tags

Overview

Protocol Buffers for the Rest of Us

Motivation

protoletariat has one goal: fixing the broken imports for the Python code generated by protoc.

Usage

Here's an example of how to use the tool, called protol:

Create a few protobuf files

// thing1.proto
syntax = "proto3";

import "thing2.proto";

package things;

message Thing1 {
  Thing2 thing2 = 1;
}

// thing2.proto
syntax = "proto3";

package things;

message Thing2 {
  string data = 1;
}

Run protoc on those files

$ mkdir out
$ protoc --python_out=out --proto_path=directory/containing/protos thing1.proto thing2.proto

Run protol on the generated code

$ protol --create-init --overwrite -g out --proto-path=directory/containing/protos thing1.proto thing2.proto

The out/thing1_pb2.py file should show a diff containing at least these lines:

-import thing2_pb2 as thing2__pb2
-
+from . import thing2_pb2 as thing2__pb2

How it works

At a high level, protoletariat converts absolute imports to relative imports.

However, it doesn't just convert any absolute import to a relative import.

The protol tool will only convert imports that were generated from .proto files. It does this by inspecting FileDescriptorProtos from the protobuf files.

The core mechanism is implemented using a simplified form of pattern matching, that looks at the Python AST, and if an import pattern is matched and corresponding rewrite rule is fired.

Help

$ protol --help
Usage: protol [OPTIONS] PROTO_FILES...

  Rewrite protoc-generated imports for use by the proletariat.

Options:
  -g, --generated-python-dir DIRECTORY
                                  Directory containing generated Python code
                                  [required]
  -p, --proto-path DIRECTORY      Protobuf file search path(s). Accepts
                                  multiple values.  [required]
  --overwrite / --no-overwrite    Overwrite all generated Python files with
                                  modified imports
  --create-init / --dont-create-init
                                  Create an __init__.py file under the
                                  `generated-python-dir` directory
  --help                          Show this message and exit.

Comments

Usage on Windows

I currently am not able to make it work on Windows. However it works in the WSL shell with the same syntax. I keep getting a FileNotFoundError: [WinError 2] on Windows. Am I doing something wrong with the syntax?

This one is not working:

protol --create-package --in-place --python-out=cosmospy_protobuf protoc --proto-path=C:\Users\felix\Documents\dev\cosmospy-protobuf\cosmospy_protobuf  C:\Users\felix\Documents\dev\cosmospy-protobuf\cosmospy_protobuf\gogoproto\gogo.proto

This one is working:

protol --create-package --in-place --python-out=cosmospy_protobuf protoc --proto-path=/mnt/c/Users/felix/Documents/dev/cosmospy-protobuf/cosmospy_protobuf  /mnt/c/Users/felix/Documents/dev/cosmospy-protobuf/cosmospy_protobuf/gogoproto/gogo.proto

Same behaviour when using subprocess.run() in a .py file:

absolute_path = os.path.abspath('cosmospy_protobuf')

#Genrated in a loop but this is how the path is generated
os.path.abspath(os.path.join(root, filename)

cmd = [sys.executable, '-m', 'protoletariat',
           '--create-package', '--in-place',
           '--python-out', 'cosmospy_protobuf',
           'protoc', f'--proto-path={absolute_path}',
           filepath]
subprocess.run(cmd)

This works on Linux but not on windows

opened by ctrl-Felix 15

fix: ensure every path is decoded using os.fsdecode

I'm using Click 7.1.2. For some reason, the input file paths were bytes instead of str, causing lots of errors when I tried to run protol.

This PR fixed the problem for me. It's probably a safe thing to merge, and it guards against other strange Click incompatibilities.

In case it matters, here's how I was invoking protol:

https://github.com/shawwn/tensorflow-checkpoint-reader/blob/c8a77dc82c88f52971f11a96e59d5ace7b595611/build.sh#L15-L16
bug

opened by shawwn 11
Generated .pyi files include __pycache__, are not idempotent
I've noticed a few things about generated .pyi files:

__pycache__ gets picked up and imported,

Entries are not sorted (though this isn't really a big deal, it kinda helps with reviewing the diff)

Entries get duplicated if run multiple times/generation is not idempotent

The last one is probably harder to solve since it appears Protoletariat is not assuming it has full control of the pyi (understandable), or else it could overwrite instead of append.
opened by lidavidm 4
chore(deps): update dependency flake8 to v6
This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | flake8 (changelog) | >=5.0.0,<6 -> >=6,<7 | | | | |

Release Notes

pycqa/flake8

v6.0.0

Compare Source

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Enabled.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Mend Renovate. View repository job log here.
dependencies
opened by renovate[bot] 3
bug: including the google.protobuf types breaks the code
Hello, Whenever I add gRPC well-known types like Empty, the protoletariat breaks the python code changing:

from google.protobuf import empty_pb2 as google_dot_protobuf_dot_empty__pb2

to

from .google.protobuf import empty_pb2 as google_dot_protobuf_dot_empty__pb2
bug
opened by bartek1912 3
chore(deps): update dependency flake8 to v6
This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | flake8 (changelog) | >=5.0.0,<6 -> >=6,<7 | | | | |

Release Notes

pycqa/flake8

v6.0.0

Compare Source

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Enabled.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Mend Renovate. View repository job log here.
dependencies
opened by renovate[bot] 2
chore(deps): update dependency mypy to ^0.930
This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | mypy (source, changelog) | ^0.920 -> ^0.930 | | | | |

Release Notes

python/mypy

v0.930

Compare Source

v0.921

Compare Source

Configuration

📅 Schedule: At any time (no schedule defined).

🚦 Automerge: Disabled due to failing status checks.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

[ ] If you want to rebase/retry this PR, click this checkbox.

This PR has been generated by WhiteSource Renovate. View repository job log here.
dependencies
opened by renovate[bot] 2
chore(deps): update dependency platformdirs to <2.6.3
This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | platformdirs | <2.6.1 -> <2.6.3 | | | | |

Release Notes

platformdirs/platformdirs

v2.6.2

Compare Source

Fix missing typing-extensions dependency.

v2.6.1

Compare Source

Add detection of $PREFIX for android.

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Enabled.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Mend Renovate. View repository job log here.
dependencies
opened by renovate[bot] 1
fix(deps): update dependency protobuf to v4
This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | protobuf | ^3.19.1 -> ^4.0.0 | | | | |

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Enabled.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

[ ] If you want to rebase/retry this PR, click this checkbox.

This PR has been generated by Mend Renovate. View repository job log here.
dependencies
opened by renovate[bot] 1
chore(flake/nixpkgs): `19e187fb` -> `836c52e6`

| SHA256 | Commit Message | | ---------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- | | fbccf6ea | python39Packages.ipywidgets: use pytestCheckHook and general cleanup | | 162d2d3a | emacs.pkgs.ement: unstable-2022-04-22 -> unstable-2022-05-05 | | c58ed2fc | python3Packages.awkward: update meta.homepage | | f02a08d4 | headset-charge-indicator: init at 2021-08-15 | | 881a8cbe | nixos/kanidm: fix release notes | | c126babb | nixos/kanidm: init | | 3b5fc1fd | kanidm: init at 1.1.0-alpha.8 | | 9af7f009 | python3Packages.azure-mgmt-msi: update propagatedBuildInputs | | 974603c9 | ecdsautils: 0.4.0 -> 0.4.1 | | 6bffe188 | python310Packages.databricks-connect: 9.1.14 -> 9.1.15 | | 1d33da66 | python310Packages.aiolimiter: init at 1.0.0 | | 38c70972 | python310Packages.azure-mgmt-resource: 21.0.0 -> 21.1.0 | | bb117f73 | python310Packages.globus-sdk: 3.7.0 -> 3.8.0 | | 321d31a4 | iwd: 1.26 -> 1.27 | | ab27f31b | home-assistant: update component-packages | | 38c5c8c0 | python310Packages.raincloudy: init at 1.1.1 | | 815f83a0 | home-assistant: update component-packages | | d2905c6f | python310Packages.pyrainbird: init at 0.4.3 | | d487bcce | bolt: 0.9.1 → 0.9.2 | | 23c4e4aa | sqlite-utils: 3.26 -> 3.26.1 | | e9f479ec | librewolf: 100.0-1 -> 100.0-2 | | 0d03ca8d | hopper: 4.5.29 -> 5.5.3 | | 0558cb24 | libffi_3_3: init | | cb7c4401 | maintainers: add armeenm | | f0255a6f | python310Packages.nbclient: run tests | | a38b583a | python310Packages.setupmeta: 3.3.0 -> 3.3.1 | | 7ec5bd9e | sift: add self to maintainers | | 763a2d7b | sift: add bash completion | | 3b26a349 | python310Packages.impacket: 0.9.24 -> 0.10.0 | | e33dbfd2 | opentabletdriver: 0.6.0.3 -> 0.6.0.4 | | 6f960f08 | home-assistant: update component-packages | | a2c236d7 | python310Packages.meater-python: init at 0.0.8 | | 58ec9c11 | python310Packages.scancode-toolkit: relax intbitset constraint | | 0f5df81d | python310Packages.intbitset: fix license | | 01853e27 | bsp-layout: fix postInstall | | ca8c8a1a | python39Packages.wandb: use the correct nbclient dependency | | 41fbaa22 | python3Packages.bond-api: 0.1.16 -> 0.1.17 | | 1dcc140d | sweethome3d: fix impurity | | 6036f050 | melody: 0.13.10 -> 0.18.0 | | f6c4cf25 | flip-link: 0.1.4 -> 0.1.6 | | f989e139 | zfs: Support zfs_force=y on the command line as well. | | 3ed4d50f | xfsprogs: 5.15.0 -> 5.16.0 | | bf59489f | liferea: 1.12.9 -> 1.13.8 | | d0b632a1 | proxysql: fix build | | abc98356 | maintainers: add zebreus | | b8b17d9b | power-profiles-daemon: 0.10.1 → 0.11.1 | | d9d51eec | libwebsockets: disable -Werror | | e0b5ba54 | nixos: Don't use grep to request ZFS credentials, and consider keystatus | | 3a71b113 | nixos: Include zfsroot in installer-systemd-stage-1 tests | | 0a161580 | zfs: Update comment for https://github.com/zfsonlinux/zfs/pull/4943 | | 8555a7fd | zfs: Allow three tries to decrypt datasets | | 44a6882f | nixos/stage-1-systemd: ZFS support | | 56ab4f61 | nixos/lxd: improve tests | | eb50502c | lucenepp: install header files of lucene-contrib | | 023a12f0 | inspircd: 3.12.0 -> 3.13.0 | | 14d54882 | libwebsockets: drop unmaintained older versions | | c517610e | swaynotificationcenter: 0.3 -> 0.5 | | 014b59a4 | umockdev: Make library path references absolute | | 35b85a12 | umockdev: 0.17.8 → 0.17.9 | | 47f0136d | neomutt: 20220415 -> 20220429 | | e6acae67 | wiki-js: 2.5.277 -> 2.5.279 | | f7ad1883 | mediainfo-gui: 21.09 -> 22.03 | | 7532dcd6 | libmediainfo: 21.09 -> 22.03 | | 3d76f7ec | input-remapper: unstable-2022-02-09 -> 1.4.2 | | a3c0afb1 | got: 0.68.1 -> 0.69 | | 4986504f | python38Packages.backports-zoneinfo: test data for zoneinfo 2022a | | 568cb2d6 | nixos/systemd/nspawn: Add missing nspawn unit options | | 0d8a3244 | AusweisApp2: 1.22.4 -> 1.22.5 | | 5a47886f | microsoft-edge: 98.0.1108.56 -> 100.0.1185.44 | | 82060bee | portfolio: 0.57.1 -> 0.57.2 | | 2f99b713 | timeular: 3.9.1 -> 4.7.1 | | 22419c93 | weechat-otr: Fix build and knownVulnerabilities | | a2b02ea6 | telegraf: 1.22.0 -> 1.22.1 | | 76591b5b | psi-plus: 1.5.1615 -> 1.5.1618 | | dcbe74f3 | protoc-gen-twirp_php: set version | | 5f63e522 | protoc-gen-twirp_php: 0.8.0 -> 0.8.1 | | 16131300 | p4v: 2020.1.1966006 -> 2021.3.2186916 |
dependencies autorebase:opt-in released

opened by phillip-ground[bot] 1
chore(flake/nixpkgs): `ab0bd3af` -> `19e187fb`

| SHA256 | Commit Message | | ---------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | | 5a51d04e | onedrive: 2.4.16 -> 2.4.17 | | b8dd32df | python310Packages.afsapi: 0.2.3 -> 0.2.4 | | 57ce8ad6 | skytemple-rust: add Foundation library on darwin | | c777cdf5 | easycrypt-runtest: init at 2022.04 | | b20934a6 | easycrypt: init at 2022.04 | | 04297436 | python310Packages.sqlmap: 1.6.4 -> 1.6.5 | | cda4885f | python310Packages.peaqevcore: 0.0.21 -> 0.0.22 | | ae75c14f | awesome-4-0: remove | | 0ab6bceb | libdeltachat: 1.78.0 -> 1.79.0 | | 55352488 | rust-analyzer: 2022-04-11 -> 2022-05-02 | | 9dd25bbe | chezmoi: 2.15.1 -> 2.15.2 | | 02900c45 | tut: 0.0.42 -> 0.0.46 | | e02a0ba3 | python310Packages.nbclient: remove comment | | c83d28b8 | lookatme: remove | | 11ca7ae7 | home-assistant: disable more dependency tests | | 5aad7233 | home-assistant: 2022.4.7 -> 2022.5.0 | | b4049c2b | python3Packages.ansible: add meta.changelog | | 3581eb92 | python3Packages.ansible: 5.7.0 -> 5.7.1 | | f886c041 | python3Packages.insteon-frontend-home-assistant: init at 0.1.0 | | 9170fcf7 | python3Packages.gcal-sync: init at 0.7.1 | | 063015ee | poetry2nix: 1.27.1 -> 1.28.0 | | 5860a9e5 | senpai: unstable-2021-12-14 -> unstable-2022-04-29 | | ec0ae174 | Revert "networkmanager-applet: rename from networkmanagerapplet" | | f7269f24 | vulkan-loader: fix cross-compilation | | 440f2e1d | python3Packages.pydeconz: 90 -> 91 | | 2401e18f | python3Packages.total-connect-client: 2022.2.1 -> 2022.3 | | db3a1b15 | python310Packages.PyChromecast: 11.0.0 -> 12.0.0 | | f58df705 | python3Packages.voluptuous: 0.13.0 -> 0.13.1 | | 8006deeb | python3Packages.zwave-js-server-python: 0.35.3 -> 0.36.1 | | fc3ed4e8 | python3Packages.zigpy: 0.44.2 -> 0.45.1 | | b1a2be80 | python3Packages.simplisafe-python: 2022.03.3 -> 2022.5.0 | | f4c6bea0 | python3Packages.pytomorrowio: 0.2.1 -> 0.3.3 | | ed5b29c1 | python310Packages.pynws: 1.3.2 -> 1.4.1 | | f5b90ecc | python310Packages.pyinsteon: 1.0.16 -> 1.1.0 | | f12a40ff | python3Packages.pyevilgenius: 1.0.0 -> 2.0.0 | | 24971206 | python310Packages.pydeconz: 87 -> 90 | | c38b06a3 | python3Packages.async-upnp-client: 0.28.0 -> 0.29.0 | | 44f8ea01 | python3Packages.androidtv: 0.0.66 -> 0.0.67 | | 9a99c652 | python3Packages.aioslimproto: 2.0.0 -> 2.0.1 | | e9476f79 | python3Packages.aioairzone: 0.3.4 -> 0.4.2 | | 9146215b | python3Packages.nbclient: 0.5.13 -> 0.6.0 | | 5f8ac4de | python3Packages.nbconvert: 6.4.5 -> 6.5.0 | | 60b4f7d8 | zerobin: raise version bounds for bleach to <6 | | 0c90ed17 | python3Packages.bleach: 4.1.0 -> 5.0.0 | | 2d5d1d4c | pssh: 2.3.1 -> 2.3.4 | | de4a99db | brave: fix GPU acceleration on Wayland | | 443e8f64 | caprine-bin: 2.55.2 -> 2.55.4 | | 166b1378 | snakemake: 7.5.0 -> 7.6.1 | | b139ab79 | qownnotes: 22.4.1 -> 22.5.0 | | ef459179 | python310Packages.pykulersky: 0.5.3 -> 0.5.4 | | eb8ce992 | ssm-session-manager-plugin: 1.2.54.0 -> 1.2.312.0 | | 2efd1090 | rich-cli: 1.5.1 -> 1.7.0 | | 3f87b172 | python3Packages.textual: 0.1.15 -> 0.1.18 | | 716b60f6 | wolfssl: 5.2.0 -> 5.3.0 | | 3c95672b | syft: 0.44.1 -> 0.45.1 | | e310b2cd | grype: 0.36.0 -> 0.36.1 | | a5151720 | python39Packages.howdoi: disable failing tests | | ab093932 | modem-manager-gui: fix build with meson >= 0.61 | | 3ea13a6f | hare: run test suite | | c3a7edf7 | purescript: drop ncurses dep | | 7d5e8244 | just: 1.1.2 -> 1.1.3 | | cc67617d | udiskie: 2.4.0 -> 2.4.2 | | ecf564b2 | cargo-sync-readme: 1.0 -> 1.1 | | c0bb20e0 | nodejs-18_x: fix completion generation | | 7b1a7987 | platformsh: 3.79.1 -> 3.79.2 | | 70f212bb | nodejs-14_x: 14.19.1 -> 14.19.2 | | 596c5e7e | Static bwa | | 47588733 | Static build for samtools | | 967a5d78 | Static builds for HTSLIB | | 0b1c28d5 | Static build for megahit | | 342e99a5 | seafile-client: build v8.0.7 from proper commit | | be8f036f | gopass: 1.14.0 → 1.14.1 | | bb7bb243 | cloudflared: 2022.4.1 -> 2022.5.0 | | 6a4134d8 | fulcio: 0.4.0 -> 0.4.1 | | b060edea | upwork: 5.6.10.7 -> 5.6.10.13 | | 37cada5b | vlc: 3.0.17 -> 3.0.17.3 | | 5aec68a7 | prometheus-aws-s3-exporter: 0.4.1 -> 0.5.0 | | c25b7497 | runitor: init at 0.9.2 | | b918d3c7 | platformsh: 3.79.0 -> 3.79.1 | | 26a47abf | python310Packages.snowflake-connector-python: 2.7.6 -> 2.7.7 | | d64b2cc1 | twister: remove | | 67b88714 | nmap: remove graphical support | | 1be5ae11 | xpf: remove | | 8ddb45c1 | caffe2: remove | | 0fd723f3 | xpdf: 4.03 -> 4.04 | | 0f1bb1f4 | jadx: 1.3.4 -> 1.3.5 | | 153b2faf | goocanvas2: switch to python3 | | f0c470f5 | oath-toolkit: Rename from oathToolkit to oath-toolkit |
dependencies autorebase:opt-in released

opened by phillip-ground[bot] 1
chore(deps): lock file maintenance
This PR contains the following updates:

| Update | Change | |---|---| | lockFileMaintenance | All locks refreshed |

🔧 This Pull Request updates lock files to use the latest dependency versions.

Configuration

📅 Schedule: Branch creation - "before 5am on monday" (UTC), Automerge - At any time (no schedule defined).

🚦 Automerge: Enabled.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Mend Renovate. View repository job log here.
dependencies
opened by renovate[bot] 0

mypy not happy with `.pyi` rewriting

This project is a great, thank you very much for making and maintaining it.

I have tried using it with typing however and I notice that mypy is not entirely happy with the rewriting of .pyi files.

I have setup an example here: https://github.com/aucampia/issue-20221116-python_protobuf_cloudevents/tree/a245d29c2b1600b6580c6db861b4b715e72b9b51

The specific .pyi file that upsets mypy is generated/protol/_plgen/example/v1/messages_pb2.pyi and the untransformed file is generated/proto/example/v1/messages_pb2.pyi

With the mypy config from the example, running mypy yields:

$ python -m poetry run mypy --show-error-codes --show-error-context 
tests/test_conflicting_ns.py:80: note: In module imported here:
generated/protol/_plgen/example/v1/messages_pb2.pyi: note: In member "field_b" of class "Something":
generated/protol/_plgen/example/v1/messages_pb2.pyi:24: error: Name "io.cloudevents.v1.cloudevents_pb2.CloudEvent" is not defined  [name-defined]
generated/protol/_plgen/example/v1/messages_pb2.pyi: note: In member "__init__" of class "Something":
generated/protol/_plgen/example/v1/messages_pb2.pyi:27: error: Name "io.cloudevents.v1.cloudevents_pb2.CloudEvent" is not defined  [name-defined]
Found 2 errors in 1 file (checked 5 source files)
make: *** [Makefile:78: python-validate-static] Error 1

The specific part of the transformation that is causing the problem is this I believe:

$ diff -u -r generated/proto/example/v1/messages_pb2.pyi generated/protol/_plgen/example/v1/messages_pb2.pyi 
--- generated/proto/example/v1/messages_pb2.pyi	2022-11-22 09:08:54.417836214 +0100
+++ generated/protol/_plgen/example/v1/messages_pb2.pyi	2022-11-22 09:08:55.062838530 +0100
@@ -5,32 +5,31 @@
 import builtins
 import google.protobuf.descriptor
 import google.protobuf.message
-import io.cloudevents.v1.cloudevents_pb2
+from ... import io
 import sys

If I change this by hand to instead do from ...io.cloudevents.v1 import cloudevents_pb2 as io_dot_cloudevents_dot_v1_dot_cloudevents__pb2 and make corresponding changes in the rest of the file to use io_dot_cloudevents_dot_v1_dot_cloudevents__pb2 instead of io.cloudevents.v1.cloudevents_pb2 then mypy is entirely happy.

opened by aucampia 0

No such option `--descriptor_set_in`
It would be great if protoletariat would pass the --descriptor_set_in parameter to protoc compiler. Now it just returns an error:

Usage: protol protoc [OPTIONS] PROTO_FILES... Try 'protol protoc --help' for help. Error: No such option: --descriptor_set_in
opened by bartek1912 1

Error while using `grpc-tools.protoc` for `protoc` subcommand

Console error:

  File "/Users/seanchok/Documents/rpc/.venv/lib/python3.10/site-packages/protoletariat/fdsetgen.py", line 137, in generate_file_descriptor_set_bytes
    subprocess.check_output(
  File "/Users/seanchok/.pyenv/versions/3.10.0/lib/python3.10/subprocess.py", line 420, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/Users/seanchok/.pyenv/versions/3.10.0/lib/python3.10/subprocess.py", line 501, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/Users/seanchok/.pyenv/versions/3.10.0/lib/python3.10/subprocess.py", line 966, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/Users/seanchok/.pyenv/versions/3.10.0/lib/python3.10/subprocess.py", line 1842, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'python -m grpc_tools.protoc'

Steps to reproduce:

install grpc-tools through pip to use protoc
generate python bindings

python -m grpc-tools.protoc \
  --proto_path=. \
  --python_out=out \
  --grpc_python_out=out \
  *.proto

Use protol protoc

protol \
  --create-package \
  --in-place \
  --python_out out
  protoc \
  --protoc-path "python -m grpc_tools.protoc" \
  --proto-path .
  *.proto

My own research:

I read through the proloteriat source code, and suspected that the subprocess.check_output call may not be able to call python to run executable modules just like that. But I'm not entirely sure because the error thrown appears to be FileNotFound.
I created a workaround by first generating the FileDescriptorSet file, and then running protol raw instead of protol protoc.

touch descriptor.temp

# generate FileDescriptorSet
python -m grpc_tools.protoc \
  --include_imports \
  --descriptor_set_out=descriptor.temp \
  --proto_path=. \
  *.proto

# run post-processing to change absolute imports generated by protoc to relative imports
protol \
  --create-package \
  --in-place \
  --python-out out \
  raw \
  descriptor.temp

I got the inspiration for the workaround after reading this section of code.

I'm not sure if protol needs to be updated to make running python -m grpc_tools.protoc possible. Or if this workaround should be made known to other people who are using the grpc-tools package as I am.

opened by 98sean98 3

Dependency Dashboard
This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

[ ] chore(deps): lock file maintenance

Detected dependencies

github-actions

.github/workflows/auto-rebase.yml

tibdex/github-app-token v1

Label305/AutoRebase v0.1

.github/workflows/ci.yml

actions/checkout v3

cachix/install-nix-action v18

cachix/cachix-action v12

actions/checkout v3

cachix/install-nix-action v18

cachix/cachix-action v12

actions/checkout v3

cachix/install-nix-action v18

cachix/cachix-action v12

actions/checkout v3

cachix/install-nix-action v18

cachix/cachix-action v12

actions/checkout v3

bufbuild/buf-setup-action v1.11.0

actions/setup-python v4

conda-incubator/setup-miniconda v2

actions/checkout v3

bufbuild/buf-setup-action v1.11.0

actions/setup-python v4

conda-incubator/setup-miniconda v2

actions/checkout v3

cachix/install-nix-action v18

cachix/cachix-action v12

actions/checkout v3

actions/checkout v3

cachix/install-nix-action v18

cachix/cachix-action v12

tibdex/github-app-token v1

actions/checkout v3

cachix/install-nix-action v18

cachix/cachix-action v12

.github/workflows/codeql-analysis.yml

actions/checkout v3

github/codeql-action v2

github/codeql-action v2

github/codeql-action v2

.github/workflows/docker.yml

actions/checkout v3

cachix/install-nix-action v18

cachix/cachix-action v12

docker/metadata-action v4

docker/login-action v2

docker/build-push-action v3

.github/workflows/update-deps.yml

actions/checkout v3

cachix/install-nix-action v18

actions/checkout v3

cachix/install-nix-action v18

tibdex/github-app-token v1

cpcloud/flake-update-action v1.0.4

poetry

pyproject.toml

astunparse >=1.6.3,<2

click >=8,<9

protobuf >=3.19.1,<5

black >=22,<24

grpc-stubs >=1.24.7,<2

mypy >=0.991,<1

mypy-protobuf >=3,<4

types-protobuf >=4.21,<5

grpcio >=1.42,<2

pytest >=7,<8

pytest-randomly >=3.10.1,<4

ruff >=0.0.198,<1

tomli >=1,<3

[ ] Check this box to trigger a request for Renovate to run again on this repository

no-op
opened by renovate[bot] 0

Releases(2.0.0)

2.0.0(Dec 28, 2022)
2.0.0 (2022-12-28)

⚠ BREAKING CHANGES

deps: click 8 is now required

deps: bump lower bound of click to 8 (113c347)

Source code(tar.gz)
Source code(zip)
protoletariat-2.0.0-py3-none-any.whl(19.59 KB)
1.1.0(Oct 29, 2022)
1.1.0 (2022-10-29)

Features

python: allow python 3.11 (1012243)

Source code(tar.gz)
Source code(zip)
protoletariat-1.1.0-py3-none-any.whl(19.61 KB)
1.0.1(Sep 11, 2022)
1.0.1 (2022-09-11)

Bug Fixes

deps: support python protobuf==4 (bd75812)

Source code(tar.gz)
Source code(zip)
protoletariat-1.0.1-py3-none-any.whl(19.60 KB)
1.0.0(Sep 11, 2022)
1.0.0 (2022-09-11)

⚠ BREAKING CHANGES

deps: Move to Python 3.8. 0.9.4 is the last version to support Python 3.7

deps: bump python lower bound to 3.8 (0480117)

Source code(tar.gz)
Source code(zip)
protoletariat-1.0.0-py3-none-any.whl(19.60 KB)
v0.9.4(Jun 22, 2022)
0.9.4 (2022-06-22)

Bug Fixes

codegen: deduplicate generated imports (e040217)

codegen: don't traverse or generate code with __pycache__ (0c083d5)

Source code(tar.gz)
Source code(zip)
protoletariat-0.9.4-py3-none-any.whl(19.61 KB)
v0.9.3(May 6, 2022)
0.9.3 (2022-05-06)

Bug Fixes

fix version number again (2865ce4)

fix version number in __init__.py (4fac8c8)

Source code(tar.gz)
Source code(zip)
protoletariat-0.9.3-py3-none-any.whl(19.17 KB)
v0.9.2(May 6, 2022)
0.9.2 (2022-05-06)

Bug Fixes

fix version number in __init__.py (fc494c9)

Source code(tar.gz)
Source code(zip)
protoletariat-0.9.2-py3-none-any.whl(19.17 KB)
v0.9.1(Apr 16, 2022)
0.9.1 (2022-04-16)

Bug Fixes

replace hyphens with suffixes when cleaning proto filename (71b9f27)

Source code(tar.gz)
Source code(zip)
protoletariat-0.9.1-py3-none-any.whl(18.81 KB)
v0.9.0(Apr 4, 2022)
0.9.0 (2022-04-04)

Features

generate init.pyi for mypy (9d7fe5b)

Source code(tar.gz)
Source code(zip)
protoletariat-0.9.0-py3-none-any.whl(18.66 KB)
v0.8.0(Apr 2, 2022)
0.8.0 (2022-04-02)

Features

buf: pass input argument through to buf (aacae63)

Source code(tar.gz)
Source code(zip)
protoletariat-0.8.0-py3-none-any.whl(18.48 KB)
v0.7.5(Feb 7, 2022)
0.7.5 (2022-02-07)

Bug Fixes

add more underscores for imports (ff4ca73)

Source code(tar.gz)
Source code(zip)
protoletariat-0.7.5-py3-none-any.whl(18.42 KB)
v0.7.4(Jan 28, 2022)
0.7.4 (2022-01-28)

Performance Improvements

build docker image with musl (8c21eb5)

Source code(tar.gz)
Source code(zip)
protoletariat-0.7.4-py3-none-any.whl(18.33 KB)
v0.7.3(Jan 4, 2022)
0.7.3 (2022-01-04)

Bug Fixes

fix release token to ensure publishing (a97e98e)

Source code(tar.gz)
Source code(zip)
protoletariat-0.7.3-py3-none-any.whl(18.33 KB)
v0.7.2(Jan 4, 2022)
0.7.2 (2022-01-04)

Bug Fixes

move to upstream semantic-release (b742a01)

Source code(tar.gz)
Source code(zip)
protoletariat-0.7.2-py3-none-any.whl(18.33 KB)
v0.7.1(Dec 24, 2021)
Documentation

Add some info (b4c9663)

Move the conda bit down (0cf67c4)

Performance

Reduce everything up to but not including musl (c21a2a9)

Source code(tar.gz)
Source code(zip)
protoletariat-0.7.1-py3-none-any.whl(18.33 KB)
protoletariat-0.7.1.tar.gz(18.16 KB)
v0.7.0(Dec 12, 2021)
Feature

Support click 7 (b994dad)

Documentation

Link to protobuf issue (2b60855)

Source code(tar.gz)
Source code(zip)
protoletariat-0.7.0-py3-none-any.whl(18.41 KB)
protoletariat-0.7.0.tar.gz(18.20 KB)
v0.6.2(Dec 4, 2021)
Fix

Ensure every path is decoded as utf-8 (a936f08)

Source code(tar.gz)
Source code(zip)
protoletariat-0.6.2-py3-none-any.whl(17.82 KB)
protoletariat-0.6.2.tar.gz(17.62 KB)
v0.6.1(Dec 2, 2021)
Documentation

Add some more information to the README.md (1d93380)

Performance

Remove the use of a dict to store rewriters (72a8fd3)

Source code(tar.gz)
Source code(zip)
protoletariat-0.6.1-py3-none-any.whl(17.79 KB)
protoletariat-0.6.1.tar.gz(17.59 KB)
v0.6.0(Dec 1, 2021)
Feature

Add the ability to bring your own (FileDescriptorSet) bytes (36589a6)

Documentation

Update documentation (3a670a0)

Source code(tar.gz)
Source code(zip)
protoletariat-0.6.0-py3-none-any.whl(17.59 KB)
protoletariat-0.6.0.tar.gz(17.13 KB)
v0.5.5(Dec 1, 2021)
Documentation

README.md: Remove redundant pypi badge (eca23d3)

Fix message about conda-forge artifacts (6b77fb2)

Replace installation lines with table (8a1a702)

Update README.md to include new options (ae2e8ea)

Performance

Only iterate through files and dependencies once (561a616)

Source code(tar.gz)
Source code(zip)
protoletariat-0.5.5-py3-none-any.whl(17.08 KB)
protoletariat-0.5.5.tar.gz(16.64 KB)
v0.5.4(Nov 29, 2021)
Fix

Add the ability to exclude google and optionally additional glob patterns from import rewrites (beadb73)

Documentation

Add docstring to _should_ignore (52225ad)

Source code(tar.gz)
Source code(zip)
protoletariat-0.5.4-py3-none-any.whl(16.56 KB)
protoletariat-0.5.4.tar.gz(15.68 KB)
v0.5.3(Nov 29, 2021)
Fix

Don't require commitlint for release (1e1a7fe)

Split out buf tests to allow for skipping without a marker (b7dabca)

Add buf marker to allow running tests if buf is known not be installable (50847d9)

Source code(tar.gz)
Source code(zip)
protoletariat-0.5.3-py3-none-any.whl(15.95 KB)
protoletariat-0.5.3.tar.gz(15.12 KB)
v0.5.2(Nov 28, 2021)
Performance

Use cpp implementation for python protobuf (ebc5614)

Source code(tar.gz)
Source code(zip)
protoletariat-0.5.2-py3-none-any.whl(15.04 KB)
protoletariat-0.5.2.tar.gz(14.93 KB)
v0.5.1(Nov 28, 2021)
Fix

Use ast replacement everywhere because typed_ast/typed_astunparse performs correct roundtrip (627bffb)

Documentation

Add installation section to README.md (edd9dfc)

Update README to include .pyi suffixes in help message (3e1ba1e)

Source code(tar.gz)
Source code(zip)
protoletariat-0.5.1-py3-none-any.whl(15.04 KB)
protoletariat-0.5.1.tar.gz(14.92 KB)
v0.5.0(Nov 26, 2021)
Feature

Add mypy and mypy grpc suffixes (e5a4b9e)

Fix

Address imports in pyi that share a prefix (87a3f61)

Distinguish between single module imports versus froms in string replacements (8107d5a)

Documentation

Add comment about why the replacement strategies are necessary (f98e41c)

Fix readme to reflect CLI (f5d264d)

Source code(tar.gz)
Source code(zip)
protoletariat-0.5.0-py3-none-any.whl(15.07 KB)
protoletariat-0.5.0.tar.gz(14.85 KB)
v0.4.0(Nov 25, 2021)
Feature

Introduce module suffixes to allow adjustment of grpc generated code (b0b644a)

Fix

Rewrite using per-file rewriters (baa883c)

Source code(tar.gz)
Source code(zip)
protoletariat-0.4.0-py3-none-any.whl(13.83 KB)
protoletariat-0.4.0.tar.gz(13.55 KB)
v0.3.1(Nov 24, 2021)
Fix

Generate packages in the output directory (a00cc06)

Fix package relative imports (ae75384)

Documentation

Put a link to the pypi package in the badge (2053313)

Use shields.io instead of fury.io (7ae2261)

Add status badges for CI and PyPI (318ba9f)

Source code(tar.gz)
Source code(zip)
protoletariat-0.3.1-py3-none-any.whl(12.54 KB)
protoletariat-0.3.1.tar.gz(12.48 KB)
v0.3.0(Nov 24, 2021)
Feature

cli: Add support for generating FileDescriptorSet blobs using buf (2d072b7)

Fix

Remove warning from collections usage (7e0847e)

Documentation

Fix typo in last "how it works" paragraph (3f8efde)

Update readme with CLI help changes (74b5648)

Fix typo (33a1505)

Source code(tar.gz)
Source code(zip)
protoletariat-0.3.0-py3-none-any.whl(11.83 KB)
protoletariat-0.3.0.tar.gz(11.58 KB)
v0.2.1(Nov 23, 2021)
Fix

Fix docker image build (ffb262a)

Documentation

Fix typo (8a32b6b)

Describe how the thing works (bb05a58)

More help (95b9f2a)

More readme (c57245b)

Source code(tar.gz)
Source code(zip)
protoletariat-0.2.1-py3-none-any.whl(10.55 KB)
protoletariat-0.2.1.tar.gz(10.07 KB)
v0.2.0(Nov 23, 2021)
Feature

Implement translation with rewrites (f7f05ce)

Initial environment setup (adafb9c)

Fix

Remove 3.9-only with_stem method call (4d7f178)

Source code(tar.gz)
Source code(zip)
protoletariat-0.2.0-py3-none-any.whl(9.52 KB)
protoletariat-0.2.0.tar.gz(8.79 KB)

Owner

Phillip Cloud

I like puns.

GitHub Repository

Protocol Buffers for the Rest of Us

Related tags

Overview

Protocol Buffers for the Rest of Us

Motivation

Usage

How it works

Help

Comments

Release Notes

Configuration

Release Notes

Configuration

Release Notes

Configuration

Release Notes

Configuration

Configuration

Configuration

Open

Detected dependencies

Releases(2.0.0)

2.0.0(Dec 28, 2022)

2.0.0 (2022-12-28)

⚠ BREAKING CHANGES

1.1.0(Oct 29, 2022)

1.1.0 (2022-10-29)

Features

1.0.1(Sep 11, 2022)

1.0.1 (2022-09-11)

Bug Fixes

1.0.0(Sep 11, 2022)

1.0.0 (2022-09-11)

⚠ BREAKING CHANGES

v0.9.4(Jun 22, 2022)

0.9.4 (2022-06-22)

Bug Fixes

v0.9.3(May 6, 2022)

0.9.3 (2022-05-06)

Bug Fixes

v0.9.2(May 6, 2022)

0.9.2 (2022-05-06)

Bug Fixes

v0.9.1(Apr 16, 2022)

0.9.1 (2022-04-16)

Bug Fixes

v0.9.0(Apr 4, 2022)

0.9.0 (2022-04-04)

Features

v0.8.0(Apr 2, 2022)

0.8.0 (2022-04-02)

Features

v0.7.5(Feb 7, 2022)

0.7.5 (2022-02-07)

Bug Fixes

v0.7.4(Jan 28, 2022)

0.7.4 (2022-01-28)

Performance Improvements

v0.7.3(Jan 4, 2022)

0.7.3 (2022-01-04)

Bug Fixes

v0.7.2(Jan 4, 2022)

0.7.2 (2022-01-04)

Bug Fixes

v0.7.1(Dec 24, 2021)

Documentation

Performance

v0.7.0(Dec 12, 2021)

Feature

Documentation

v0.6.2(Dec 4, 2021)

Fix

v0.6.1(Dec 2, 2021)

Documentation

Performance

v0.6.0(Dec 1, 2021)

Feature

Documentation

v0.5.5(Dec 1, 2021)

Documentation