Implement custom protoc plugin to generate OTLP JSON class definitions by herin049 · Pull Request #4910 · open-telemetry/opentelemetry-python

herin049 · 2026-02-11T23:18:12Z

Description

This PR implements a custom protoc plugin to automatically generate OTLP JSON class definitions. Consensus was reached deciding that a custom protoc plugin would be the best route for adding support for OTLP JSON while minimizing the number of runtime dependencies (for additional context reference #4886). This PR is a refined version of the first part of a set of changes that are in #4902. Please view the draft PR for an overview of how these packages will be used.

Fixes #1003

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

How Has This Been Tested?

tox run -e $(tox --listenvs | grep opentelemetry-protojson | tr '\n' ',')

Does This PR Require a Contrib Repo Change?

Yes. - Link to PR:
No.

Checklist:

Followed the style guidelines of this project
Changelogs have been updated
Unit tests have been added
Documentation has been updated

mmanciop · 2026-02-13T10:53:16Z

I am very much looking forward to this landing: protobuf is a pretty toxic dependency when automatically injecting Python SDKs with the OpenTelemetry Injector

codegen/opentelemetry-codegen-json/src/opentelemetry/codegen/json/analyzer.py

pmcollins · 2026-02-17T19:38:44Z

I think it would be valuable to add a test that validates the generated JSON by feeding it to google.protobuf.json_format.Parse. Maybe you could then compare dictionary representations of the original object with that of the resulting proto object for equality.

herin049 · 2026-02-17T23:58:47Z

I think it would be valuable to add a test that validates the generated JSON by feeding it to google.protobuf.json_format.Parse. Maybe you could then compare dictionary representations of the original object with that of the resulting proto object for equality.

Just to make sure I understand your suggestion properly, currently I have an integration test which runs the plugin to generate the ProtoJSON Python classes and tests the serialization/deserialization. Are you suggesting that we also generate the corresponding Protobuf definitions and then perform the JSON serialization using the ProtoJSON classes, load the JSON using the Protobuf definitions and then compare the attributes on the ProtoJSON and Protobuf classes?

herin049 · 2026-02-18T03:53:07Z

@pmcollins Since we mostly care about serialization for the SDK, I added tests to use google.protobuf.json_format.MessageToDict and verify that the contents are identical. Let me know what you think, and if there are another other test cases that might be useful to add.

pmcollins · 2026-02-18T15:08:17Z

Thanks @herin049. My suggestion is to simulate a Collector receiving JSON. In other words, end-to-end tests that create SDK objects typically sent to an exporter for emission (traces, metrics, and logs), serialize those objects into JSON using the functionality in this PR, send that JSON to our simulated Collector (google.protobuf.json_format.Parse), and make sure that the "Collector" produces data that matches what was fed into the other end of the pipe.

herin049 · 2026-02-18T15:33:27Z

@pmcollins I will be adding the opentelemetry-exporter-otlp-json-common and opentelemetry-exporter-otlp-json-http packages in a follow up PR to limit the size of this PR. The opentelemetry-exporter-otlp-json-common will be analogous to the existing opentelemetry-exporter-otlp-proto-common package and will include the logic to convert SDK objects to the ProtoJSON objects in the generated opentelemetry-proto-json package added in this PR. Do we want to add the roundtrip tests you are describing into the opentelemetry-exporter-otlp-proto-common package, and defer this to the next PR? I am open to including them in this PR as well, just not sure where exactly we would want these tests to live. Feel free to look at the draft PR #4902 for additional context.

pmcollins · 2026-02-19T15:17:47Z

Makes sense @herin049 -- can be in a subsequent PR.

herin049 · 2026-02-27T15:52:36Z

@aabmass I have rewritten the commit history to separate out the addition of the opentelemetry-codegen-json package an the auto generated opentelemetry-proto-json package with a final commit to integrate these packages into the rest of the repo. Hopefully this makes reviewing a bit easier, let me know if you would prefer that I break down these changes further.

aabmass

It's a bit large LGTM. I will look over the generated code too

codegen/opentelemetry-codegen-json/README.rst

aabmass · 2026-03-05T17:55:06Z

codegen/opentelemetry-codegen-json/src/opentelemetry/codegen/json/runtime/json_codec.py

+def encode_repeated(
+    values: typing.Optional[list[typing.Any]],
+    map_fn: typing.Callable[[typing.Any], typing.Any],
+) -> list[typing.Any]:


nit use generic

def encode_repeated( values: list[T] | None, map_fn: typing.Callable[[T], typing.Any], Comment view) -> list[typing.Any]:

aabmass · 2026-03-05T18:10:03Z

codegen/opentelemetry-codegen-json/src/opentelemetry/codegen/json/plugin.py

+
+@contextmanager
+def code_generation() -> Iterator[
+    Tuple[plugin.CodeGeneratorRequest, plugin.CodeGeneratorResponse],


nit tuple[...]

aabmass · 2026-03-05T18:20:24Z

codegen/opentelemetry-codegen-json/src/opentelemetry/codegen/json/types.py

+    if proto_type in INT64_TYPES:
+        return True
+    return proto_type in {


I think this can be a single bool expression return ... or ...

aabmass · 2026-03-05T18:25:17Z

codegen/opentelemetry-codegen-json/src/opentelemetry/codegen/json/writer.py

This code LGTM but I would be open to using some off-the-shelf strongly typed lib for this too. Up to you, I'm not sure if something like that exists.

codegen/opentelemetry-codegen-json/src/opentelemetry/codegen/json/generator.py

aabmass · 2026-03-05T18:33:34Z

codegen/opentelemetry-codegen-json/tests/conftest.py

+        ]
+    )
+
+    sys.path.insert(0, str(gen_path.absolute()))


this should be cleaned up after the test, maybe use https://docs.pytest.org/en/stable/reference/reference.html#pytest.MonkeyPatch.syspath_prepend

aabmass · 2026-03-05T18:34:44Z

codegen/opentelemetry-codegen-json/tests/test_serdes.py

+
+
+def test_unknown_fields_ignored(
+    test_v1_types: tuple[Type[Any], Type[Any]],


type[Any]

aabmass · 2026-03-05T18:37:36Z

codegen/opentelemetry-codegen-json/src/opentelemetry/codegen/json/runtime/__init__.py

Oh, does this runtime module need to be available to the generated code as a dependency? I.e. they need to install opentelemetry-codegen-json?

aabmass · 2026-03-05T18:50:18Z

opentelemetry-proto-json/src/opentelemetry/proto_json/collector/logs/v1/logs_service.py

+    def to_json(self) -> builtins.str:
+        """
+        Serialize this message to a JSON string.
+
+        Returns:
+            JSON string
+        """
+        return json.dumps(self.to_dict())


Would it be reasonable to put this in a base class to try to reduce the code size a bit? Maybe there are downsides, lmk

I've opted to add a @json_serde class decorator

aabmass · 2026-03-05T18:52:32Z

opentelemetry-proto-json/src/opentelemetry/proto_json/collector/logs/v1/logs_service.py

+        """
+        return json.dumps(self.to_dict())
+
+    @builtins.classmethod


Since it's not super obvious to me, what's the need of using builtins? To avoid accidental name clashes?

Yes, I used this to prevent name clashes. It might be overkill, but I figured it wouldn't hurt. This is also the convention that the mypy protobuf library used which I took inspiration from.

aabmass · 2026-03-05T18:59:02Z

pyproject.toml

 typeCheckingMode = "standard"
 pythonVersion = "3.9"

 include = [


Can we typecheck the plugin code here, and ideally even the generated code as a sanity check?

MikeGoldsmith

Looks good to me outside of what @aabmass raised. Found one minor typo 👍🏻

MikeGoldsmith · 2026-03-06T11:56:10Z

codegen/opentelemetry-codegen-json/src/opentelemetry/codegen/json/types.py

+    Args:
+        snake_name: The field name in snake_case.
+    Returns:
+        The field name converted to camelCase.k


Suggested change

The field name converted to camelCase.k

The field name converted to camelCase

herin049 requested a review from a team as a code owner February 11, 2026 23:18

herin049 force-pushed the feat/protoc-otlp-json-plugin branch from f66380d to a9d10c7 Compare February 12, 2026 04:09

xrmx added this to Python PR digest Feb 12, 2026

xrmx moved this to Ready for review in Python PR digest Feb 12, 2026

aabmass reviewed Feb 17, 2026

View reviewed changes

codegen/opentelemetry-codegen-json/src/opentelemetry/codegen/json/analyzer.py Outdated Show resolved Hide resolved

herin049 force-pushed the feat/protoc-otlp-json-plugin branch from 4d4119c to 4bce86a Compare February 18, 2026 03:49

herin049 requested a review from aabmass February 24, 2026 17:29

herin049 added 3 commits February 27, 2026 10:48

add opentelemetry-codegen-json package

49d2c70

add opentelemetry-proto-json package

81a86b0

update project files to incorporate new packages

05b459f

herin049 force-pushed the feat/protoc-otlp-json-plugin branch from 4bce86a to 05b459f Compare February 27, 2026 15:49

aabmass approved these changes Mar 5, 2026

View reviewed changes

aabmass reviewed Mar 5, 2026

View reviewed changes

MikeGoldsmith reviewed Mar 6, 2026

View reviewed changes

herin049 added 3 commits March 6, 2026 23:16

add 'Private :: Do Not Upload' to codegen pyproject.toml classifiers

b6f30e7

add @json_serde decorator and update runtime utility file name

1b6ec6b

update docstring

ffa7a16



		def test_unknown_fields_ignored(
		test_v1_types: tuple[Type[Any], Type[Any]],

	The field name converted to camelCase.k
	The field name converted to camelCase

Conversation

herin049 commented Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

How Has This Been Tested?

Does This PR Require a Contrib Repo Change?

Checklist:

Uh oh!

mmanciop commented Feb 13, 2026

Uh oh!

Uh oh!

pmcollins commented Feb 17, 2026

Uh oh!

herin049 commented Feb 17, 2026

Uh oh!

herin049 commented Feb 18, 2026

Uh oh!

pmcollins commented Feb 18, 2026

Uh oh!

herin049 commented Feb 18, 2026

Uh oh!

pmcollins commented Feb 19, 2026

Uh oh!

herin049 commented Feb 27, 2026

Uh oh!

aabmass left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aabmass Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MikeGoldsmith left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

herin049 commented Feb 11, 2026 •

edited

Loading

aabmass Mar 5, 2026 •

edited

Loading