(pulsar) branch master updated: [improve][pip] PIP-246: Improved PROTOBUF_NATIVE schema compatibility checks without using avro-protobuf (#26695)

[pe...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

codelipenghui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/master by this push:
     new e7d35a02142f [improve][pip] PIP-246: Improved PROTOBUF_NATIVE schema compatibility checks without using avro-protobuf (#26695)
e7d35a02142f is described below

commit e7d35a02142fd101b2e56ca5df0c9b2ea33c5da0
Author: sinan liu <li...@gmail.com>
AuthorDate: Thu Oct 15 02:23:13 2026 +0800

    [improve][pip] PIP-246: Improved PROTOBUF_NATIVE schema compatibility checks without using avro-protobuf (#26695)
---
 pip/000-INDEX.md |   1 +
 pip/pip-246.md   | 593 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 594 insertions(+)

diff --git a/pip/000-INDEX.md b/pip/000-INDEX.md
index 9f647d246026..c21ba789432e 100644
--- a/pip/000-INDEX.md
+++ b/pip/000-INDEX.md
@@ -172,6 +172,7 @@ The following table lists all current PIPs in this directory, sorted by PIP numb
 | PIP-205 | [Reactive Java client for Apache Pulsar](pip-205.md) |
 | PIP-209 | [Separate C++/Python clients to own repositories](pip-209.md) |
 | PIP-243 | [Register Jackson Java 8 support modules by default](pip-243.md) |
+| PIP-246 | [Improve PROTOBUF_NATIVE schema compatibility checks without avro-protobuf](pip-246.md) |
 | PIP-249 | [Pulsar website redesign](pip-249.md) |
 | PIP-259 | [Make the config httpMaxRequestHeaderSize of the pulsar web server to configurable](pip-259.md) |
 | PIP-261 | [Restructure Getting Started section](pip-261.md) |
diff --git a/pip/pip-246.md b/pip/pip-246.md
new file mode 100644
index 000000000000..5b0cb3a9b558
--- /dev/null
+++ b/pip/pip-246.md
@@ -0,0 +1,593 @@
+# PIP-246: Improve PROTOBUF_NATIVE schema compatibility checks without avro-protobuf
+
+# Background knowledge
+
+Pulsar's schema registry stores schema versions for a topic and checks whether a proposed schema
+can coexist with previously registered versions. A compatibility strategy determines the direction
+of the check. `BACKWARD` asks whether a new reader can read data written with the previous schema;
+`FORWARD` asks whether the previous reader can read data written with the new schema; `FULL` requires
+both. Their transitive variants compare against all versions selected by the registry, rather than
+only the latest version.
+
+`PROTOBUF_NATIVE` stores a Protobuf schema as a serialized `FileDescriptorSet`, the root file name,
+and the root message's fully qualified name. The descriptor set contains the message definitions and
+their dependencies. The broker can reconstruct these descriptors without loading application classes
+or converting the schema to Avro. This representation is distinct from Pulsar's legacy `PROTOBUF`
+schema type.
+
+Protobuf binary messages identify fields by number and encoding, rather than by field name.
+Compatibility also depends on required fields, repeated fields, message types, enums, maps, and
+`oneof` groups. A `oneof` allows only one of its member fields to be set at a time. A closed enum
+exposes only its declared numeric values through the field; an open enum can also represent
+unrecognized numeric values.
+
+Protobuf supports `proto2`, `proto3`, and Editions. Editions express behaviors such as field presence,
+enum openness, and repeated-field encoding through inherited feature settings. Consequently, comparing
+syntax strings or raw field labels is insufficient to determine compatibility. Further background is
+available in the [Protobuf language guide](https://protobuf.dev/programming-guides/proto3/#updating)
+and [Editions feature documentation](https://protobuf.dev/editions/features/).
+
+The existing `PROTOBUF_NATIVE` compatibility checker requires equal root message full names for the
+six directional strategies. It does not compare fields. Checkers are selected through the broker's
+`schemaRegistryCompatibilityCheckers` configuration, with one checker per schema type.
+
+# Motivation
+
+Two schemas can retain the same root message name while making incompatible changes to their fields.
+For example, changing `Order.id` from `int64` to `string` at field number `1` passes the root-name
+check. The schemas describe different encodings, and a reader may no longer obtain the expected
+field value.
+
+Even a successful parse does not establish safe evolution. Changing an integer field's width can
+truncate values, and combining independent fields into a `oneof` can discard a value. The registry
+does not know application value restrictions or the order in which every producer and consumer will
+be upgraded, so it cannot establish the additional conditions needed to permit such changes.
+
+This proposal introduces field-level compatibility checks using Protobuf descriptors directly,
+while retaining the existing checker as the default. It permits ordinary field additions without
+an explicit default and changes between packed and expanded representations of the same repeated
+scalar field. It rejects unsafe changes and applies a conservative policy to some changes that
+Protobuf can read, including lossless type widening in one direction.
+
+# Goals
+
+## In Scope
+
+- Provide an opt-in broker checker for `PROTOBUF_NATIVE`, including native schemas used as the key
+  or value of a `KEY_VALUE` schema.
+- Define directional binary reading rules for fields reachable from the root message, including
+  nested and recursive messages, required fields, maps, real `oneof` groups, and enums.
+- Support `proto2`, `proto3`, Edition 2023, and Edition 2024 within the feature boundaries below.
+- Reuse the existing descriptor storage format, schema registry strategies, and checker extension
+  point, without Avro conversion or modifications to upstream Protobuf.
+- Provide bounded diagnostics that identify incompatible changes and unsupported constructs.
+- Preserve the existing checker as the default and support coordinated activation and rollback.
+
+## Out of Scope
+
+- Compatibility of generated source APIs, ProtoJSON, text format, RPC services, or application
+  semantics such as units and business meaning.
+- Validation of each published message, preservation of every unknown field through arbitrary
+  read/write cycles, or a guarantee of lossless whole-message round trips.
+- Client Protobuf runtime upgrades, generated-class compatibility, shading, and changes to the
+  legacy `PROTOBUF` schema type.
+- Changes to the registry's consumer compatibility strategy mapping.
+- A new schema storage format, wire protocol, public validator SPI, or external registry dependency.
+- Repairing existing schema histories, automatically migrating topics, or changing the default
+  checker in this PIP.
+- Extension payload compatibility, MessageSet, and schemas of dynamically embedded payloads that
+  are not identified by the root descriptor graph.
+
+# High Level Design
+
+Introduce an advanced checker as an opt-in alternative to the existing root-name checker. Operators
+will select it through `schemaRegistryCompatibilityCheckers`. The two checkers are alternatives;
+only one will handle `PROTOBUF_NATIVE` in a broker.
+
+The registry will continue to select the historical schemas to compare. The advanced checker will
+reconstruct their root descriptors and interpret each comparison as a writer and a reader. It will
+compare fields by number, follow referenced message and enum types, and apply the directional rules
+below. A comparison will either succeed or identify an incompatible change or unsupported construct.
+
+```mermaid
+flowchart TD
+    A[Schema registration or compatibility request] --> B[Existing registry policy and history selection]
+    B --> C[Configured checker for PROTOBUF_NATIVE]
+    C --> D[Root-name checker by default]
+    C --> E[Advanced checker when explicitly selected]
+    E --> F[Map strategy to writer and reader]
+    F --> G[Compare root identity and reachable fields]
+    G --> H[Accept or return an incompatibility diagnostic]
+```
+
+The proposal changes schema compatibility decisions at registration and compatibility-check time.
+It adds no broker-side validation to message publication or dispatch. JSON, Avro, External, and
+deployment-specific checkers will retain their existing compatibility rules.
+
+# Detailed Design
+
+## Design & Implementation Details
+
+Define `canRead(writer, reader)` as whether values written in the declared fields of a valid writer
+message can be read according to the reader under the rules in this proposal. Writer messages are
+assumed to satisfy the writer's required-field and `oneof` constraints. Arbitrary raw bytes or unknown
+fields carried forward from another schema are not evidence of what the writer descriptor guarantees.
+
+This is a conservative policy for binary reading. A schema may add or remove ordinary fields, so
+acceptance does not promise that every field remains visible to every application. For fields present
+on both sides, the checker will reject the unsafe conversions and combinations defined below. Root
+identity, default-value stability, and detectable renumbering are additional Pulsar policy constraints.
+All scalar type changes will be rejected. This includes lossless directional widenings such as an
+`int32` writer with an `int64` reader, as well as conversions whose safety depends on values or upgrade
+order. Type identity is an explicit policy choice, not a claim that every rejected direction loses data.
+
+The baseline is Protobuf binary schema semantics, with runtime-dependent enum and UTF-8 behavior
+interpreted according to the Java runtime bundled with Pulsar. Acceptance will not certify every
+client language or runtime, nor establish application-level semantic compatibility.
+
+The supported language set will be `proto2`, `proto3`, Edition 2023, and Edition 2024. Support requires
+that the broker can reconstruct the descriptors and resolve the features used by these rules.
+A dependency upgrade will not automatically expand this set. A later Edition or an uninterpretable
+feature will produce `UNSUPPORTED_FEATURE` when a comparison is required.
+
+The comparison scope starts at the declared root message and follows field references, including map
+values and imported message and enum types. Root messages may themselves be nested. When a
+non-identical pair is compared, both reachable graphs must satisfy the supported-feature boundary,
+including fields without a match in the other schema. Feature settings inherited from containing
+messages also apply, even when those containing messages are not otherwise reachable through fields.
+
+Unreachable declarations will impose no additional evolution rules. The root file and its imported
+dependencies must nevertheless be valid for descriptor reconstruction. Source comments, services,
+and known options that do not affect binary reading will not constrain compatibility. Unknown feature
+extensions will remain subject to the supported-feature boundary.
+
+| Construct | Proposed behavior |
+| --- | --- |
+| Ordinary fields, nested messages, recursion, maps, and real `oneof` groups | Apply the directional rules below. |
+| Groups and Editions delimited messages | Compare the effective message encoding and the contents. Reject a change between delimited and length-prefixed encoding. |
+| Reachable messages with extension ranges or MessageSet encoding | Return `UNSUPPORTED_FEATURE`; ordinary-field comparison cannot establish compatibility of their complete payloads. |
+| Enum fields whose Java legacy closed-enum behavior differs from the declared enum behavior | Return `UNSUPPORTED_FEATURE`. |
+| `google.protobuf.Any` | Compare its declared wrapper fields without resolving `type_url` or inspecting the embedded payload. |
+| Application data encoded inside `bytes` or `string` | Compare the declared field only. |
+| Ordinary custom options | Do not infer application validation or serialization behavior from them. |
+| Unknown feature fields or extensions, including unknown fields within Java feature options | Return `UNSUPPORTED_FEATURE`. |
+
+Descriptor reconstruction will use the existing stored representation and preserve the standard Java
+feature options needed to determine binary reading behavior. For files with a package, a stored root
+name without the `<package>.` prefix will use legacy lookup first, followed by exact package-relative
+lookup if legacy lookup fails. This will preserve which message an existing alias selects, even when
+the alias also matches a different relative path. Fully qualified root names and names in files without
+a package, including nested roots, will use exact lookup first, with legacy lookup as a fallback.
+For example, `rootMessageTypeName=Order` in a root file with package `example` will remain readable as
+`example.Order`. The root-identity comparison will use the resolved descriptor's full name. Newly
+serialized Java schemas will continue to store that full name; existing envelopes will not be rewritten.
+
+Missing root files, unresolved or cyclic file imports, and root names that cannot resolve to a message
+will produce explicit reconstruction errors. Recursive message references remain valid and are
+distinct from cyclic file imports. These changes will apply to the shared native deserializer used by
+the default checker, schema upload validator, and client generic readers, as well as the advanced
+checker. They will not enable field-level evolution rules in the default checker. Preserving Java
+features can affect client decoding, including effective UTF-8 validation, regardless of checker choice.
+
+For schema registration or a general compatibility request comparing an existing schema `S` with
+a proposed schema `N`, the required directions will be:
+
+| Strategy | Required comparison |
+| --- | --- |
+| `BACKWARD` | `canRead(S, N)` |
+| `FORWARD` | `canRead(N, S)` |
+| `FULL` | Both directions |
+| `BACKWARD_TRANSITIVE` | `canRead(Si, N)` for every historical `Si` selected by the registry |
+| `FORWARD_TRANSITIVE` | `canRead(N, Si)` for every selected `Si` |
+| `FULL_TRANSITIVE` | Both directions for every selected `Si` |
+
+History selection and deleted-history handling will remain the registry's responsibility. The checker
+will compare the supplied schemas without fetching additional history or inferring a latest version
+from collection order. An empty selected history requires no directional comparison.
+
+For the six directional strategies, an existing/proposed pair with byte-identical native schema data
+will be accepted before descriptor reconstruction and supported-feature validation. This is reuse of
+an existing definition, not certification that its features are supported for evolution. Each other
+selected historical version will still be compared. Equality of descriptors after parsing, JSON
+formatting differences, or matching a single version in a transitive history will not bypass those
+other comparisons. `ALWAYS_INCOMPATIBLE` will retain its rejection when the checker is invoked.
+
+`ALWAYS_COMPATIBLE` will bypass evolution constraints without disabling existing schema-data validation
+during registration. `ALWAYS_INCOMPATIBLE` will reject evolution when a comparison is required.
+An unexpected strategy will be rejected explicitly. First registration and reuse of an already
+registered schema will retain their existing shortcuts; enabling the checker will not certify every
+stored schema. A `KEY_VALUE` schema will use the configured checker for each component's schema type,
+with the direction and history selected by the registry.
+
+Consumer compatibility will retain the existing strategy mapping. Let `C` be the consumer schema,
+`L` the latest active schema, and `Si` each active historical schema selected by the registry:
+
+| Configured strategy | Strategy passed to the checker | Selected history | Required comparison |
+| --- | --- | --- | --- |
+| `BACKWARD` | `BACKWARD` | Latest | `canRead(L, C)` |
+| `FORWARD` | `BACKWARD` | Latest | `canRead(L, C)` |
+| `FULL` | `BACKWARD` | Latest | `canRead(L, C)` |
+| `BACKWARD_TRANSITIVE` | `BACKWARD_TRANSITIVE` | All selected versions | `canRead(Si, C)` for every `Si` |
+| `FORWARD_TRANSITIVE` | `BACKWARD` | Latest | `canRead(L, C)` |
+| `FULL_TRANSITIVE` | `FULL_TRANSITIVE` | All selected versions | Both `canRead(Si, C)` and `canRead(C, Si)` for every `Si` |
+| `ALWAYS_COMPATIBLE` | Checker bypassed | None | No evolution check |
+| `ALWAYS_INCOMPATIBLE` | `ALWAYS_INCOMPATIBLE` | All selected versions | Reject when the checker is invoked |
+
+This mapping includes a policy restriction beyond consumer readability: `FULL_TRANSITIVE` requires
+the reverse direction as well, while `FULL` checks only whether the consumer can read the latest
+schema. `FORWARD_TRANSITIVE` also retains its latest-only consumer check.
+
+For example, suppose every selected producer schema declares `required int32 id = 1`, and a consumer
+declares `optional int32 id = 1` with the same effective default. The consumer can read those producer
+messages. It passes the required-field rule under consumer `FULL` and `BACKWARD_TRANSITIVE`, but fails
+under `FULL_TRANSITIVE`: a hypothetical writer using `C` could omit `id`, so `canRead(C, Si)` fails.
+This enforces the retained registry policy rather than indicating that the consumer cannot read the
+producer's messages.
+
+Existing registry shortcuts and missing-schema behavior will remain at their current boundaries.
+The advanced checker will additionally skip byte-identical native pairs in transitive consumer checks.
+A consumer with an unchanged schema can nevertheless be refused when another selected version is
+incompatible or uses unsupported features. This matters after the restart required to activate the
+checker: schema-bearing subscriptions other than `AUTO_CONSUME` will be checked again on reconnection.
+Changing consumer strategy mapping is outside this proposal because it would affect registry semantics
+shared with other schema types.
+
+For each writer/reader message pair, fields will be matched by number within their containing messages.
+The following rules will apply:
+
+| Condition | Result |
+| --- | --- |
+| Root message full names differ | Reject, preserving the Pulsar root identity constraint. |
+| A writer field has no corresponding reader field | Allow it to be unknown to the reader. |
+| A reader field has no corresponding writer field | Allow if the reader field is not required. No explicitly declared default is necessary. |
+| A reader field is required | The corresponding writer field must also be required, and the other rules must pass. |
+| A field keeps its number but changes its name | Allow if the other rules pass. |
+| A field keeps its name but changes its number within the corresponding message | Reject as a detectable field-number change. |
+| A matched field changes its scalar type, or changes between scalar, enum, and message kinds | Reject, including conversions that share a wire type. |
+| A matched field changes between singular and repeated | Reject in either direction. |
+| A packable repeated field keeps its type but changes packed/expanded encoding | Allow. |
+| A matched singular scalar or enum field changes its effective default value | Reject as a conservative policy constraint. |
+| A matched field refers to a message | Compare its effective encoding and the referenced message pair. |
+| A matched field is a map | Both sides must be maps with the same key type; apply the scalar, enum, or message rules to the value. |
+| A map changes to or from a repeated entry message | Reject, even if the wire representation is similar. |
+| A matched string field adds UTF-8 validation that the writer does not guarantee | Reject. Removing that validation does not by itself reject the direction. |
+| A field changes between implicit and explicit presence without becoming required | Allow if the default, enum, `oneof`, and other field rules pass. |
+
+For referenced message and enum types, a type-name change alone will not reject a comparison. Their
+structures and effective behaviors will determine compatibility. The full-name identity requirement
+applies specifically to the root message.
+
+Default comparison will use the effective value, regardless of whether it was explicitly declared.
+For example, an explicit numeric default of `0` and the same implicit default are equivalent. Enum
+defaults will be compared by number, bytes by contents, and floating-point defaults by a comparison
+that treats NaN defaults consistently and distinguishes positive and negative zero. Repeated and
+message fields will not receive scalar default-value comparisons.
+
+Declaration reordering will be allowed when effective behavior is preserved. For example, changing
+`enum State { A = 1; B = 2; }` to `enum State { B = 2; A = 1; }` in proto2 changes the implicit default
+of `optional State state = 1` from `1` to `2`. An absent field then yields a different value, so the
+default-value rule rejects the comparison. If both fields explicitly retain `[default = A]`, their
+effective default remains `1`, and the reordering alone is allowed.
+
+Requiredness will be directional. Changing an optional field to required fails `BACKWARD` because
+the previous writer may omit it, but passes this rule under `FORWARD` because the proposed writer
+guarantees it. `FULL` fails. Reversing the change reverses the directional results. The same rule
+applies to Editions `LEGACY_REQUIRED`.
+
+For each real reader `oneof`, consider its member field numbers also present in the writer. If there
+are two or more, they must belong to the same real writer `oneof`. Otherwise the writer can supply
+multiple fields that the reader would make mutually exclusive. This permits moving one independent
+field into a new `oneof`, or splitting a writer `oneof` in the reader, when the remaining rules pass.
+The reverse direction can fail. Renaming a `oneof` does not change its exclusivity constraints.
+Synthetic `oneof` declarations used to represent proto3 optional presence will be excluded from
+this relationship rule.
+
+Enum comparison will use numeric values, deduplicating aliases. Let `W` and `R` be the sets of declared
+numbers in the writer and reader enums:
+
+| Writer behavior | Reader behavior | Rule |
+| --- | --- | --- |
+| Closed | Closed | Require `W` to be a subset of `R`. |
+| Closed | Open | Allow, subject to the default-value and other field rules. |
+| Open | Open | Allow, subject to the other rules; unrecognized numeric values remain representable. |
+| Open | Closed | Reject: the writer may produce a number absent from `R`, even if the declared sets are equal. |
+
+The numeric-set rule applies to the full signed 32-bit enum range. Renaming a value or adding an alias
+for the same number does not change binary interpretation. Changing the number assigned to an
+existing named value will be rejected as detectable renumbering. Moving an unrecognized value into
+an unknown-field set is insufficient to establish readability: it can hide a singular value, remove
+repeated elements, or remove map entries.
+
+A reachable enum field will be unsupported if its Java legacy closed-enum behavior differs from the
+declared openness of its enum. This includes the legacy treatment of a proto3 enum referenced by a
+proto2 field and an Editions Java override that creates the same mismatch. This restriction applies
+to either graph even when the field is unchanged or has no match in the other schema. It avoids
+assuming agreement between conflicting runtime interpretations; it does not establish that the
+proposed field change itself would prevent reading.
+
+Compatibility will use effective feature settings after inheritance, rather than syntax strings or
+raw labels alone. Relevant settings include field presence, enum openness, repeated-field encoding,
+UTF-8 validation, and message encoding. Feature settings on files, messages, fields, `oneof` groups,
+enums, and enum values will participate where applicable, including enclosing scopes of referenced
+nested types.
+
+Unknown fields and extensions within feature metadata will be rejected, including unknown numeric
+feature values and unknown fields inside Java feature options. Explicit `*_UNKNOWN` values for
+`field_presence`, `enum_type`, `repeated_field_encoding`, `utf8_validation`, and `message_encoding`
+will also be rejected. The same applies to Java `features.(pb.java).utf8_validation`. An omitted
+setting will instead inherit its enclosing or Edition default. This explicit-value restriction
+concerns the listed binary features; it will not impose the same rule on known JSON, naming, or
+code generation settings that do not affect binary reading.
+
+This proposal deliberately keeps unregistered language feature extensions outside the supported set.
+For example, a shared Editions schema containing `features.(pb.go).api_level` or
+`features.(pb.cpp).string_type` will be unsupported when a non-identical pair requires comparison,
+even though these code-generation settings do not determine Java payload parsing. The checker will
+not silently assume that every unknown extension is irrelevant to binary reading. Supporting an
+explicit allowlist of other-language extensions can be considered separately.
+
+For a matched string field, effective UTF-8 validation can depend on proto2 `java_string_check_utf8`,
+Editions `features.utf8_validation`, or Java `features.(pb.java).utf8_validation`. A writer without
+validation can produce bytes rejected by a validating reader, so enabling validation in the reader
+will reject that direction. The reverse direction will remain subject to the other field rules.
+
+Several rules are intentionally stricter than
+[Protobuf's binary evolution guidance](https://protobuf.dev/programming-guides/proto3/#updating):
+
+| Change that Protobuf may read | Pulsar policy |
+| --- | --- |
+| Lossless widening such as `int32` to `int64`, `uint32` to `uint64`, or `sint32` to `sint64` in the reading direction | Reject all scalar type changes. |
+| Singular/repeated changes that a parser can consume | Reject in both directions rather than rely on last-value or merge behavior. |
+| A changed effective default with unchanged field encoding | Reject because the value observed for an absent field changes. |
+| Reusing a field name under a new number, or changing a named enum value's number | Reject detectable renumbering rather than treat it as unrelated removal and addition. |
+| A map and its corresponding repeated entry message | Reject changes between them because map semantics can discard duplicate keys or ordering. |
+
+These are policy rejections rather than claims that parsing necessarily fails for every such change.
+A descriptor pair cannot reliably distinguish a rename combined with renumbering from an intentional
+deletion and addition, or detect changed application meaning at an unchanged number. Authors should
+reserve removed field numbers and avoid reusing them. Transitive checks can detect conflicting
+definitions in older versions that latest-only checks miss.
+
+Recursive and mutually recursive message graphs will be compared with request-local state. A
+descriptor may participate in multiple ordered writer/reader pairs, each with its own obligations.
+Both pending and completed message pairs will be memoized for the whole comparison, so shared
+subgraphs will not be walked again. Completed enum pairs and sorted field/oneof lists will also be
+reused; many fields sharing one enum must not repeatedly rebuild its value sets.
+
+Pair memoization alone does not provide an adequate work limit: writer and reader cycles of coprime
+lengths `n` and `m` can require `n * m` distinct pairs. Each selected historical version and direction
+will therefore have a deterministic work budget. Work accounting will cover graph validation,
+enqueued message pairs, examined fields and oneof members, and enum values. Initial graph validation
+will have a ceiling of 100,000 work units. If it uses `G` units, the total validation and comparison
+budget will be `min(100000, 16 * G)`. Field and oneof ordering will be prepared once per descriptor,
+with the elements charged before allocating the sorted lists.
+
+Exhausting the budget will reject the comparison with `COMPARISON_LIMIT_EXCEEDED`, even if the
+schemas might be compatible after more work; it will never fall back to acceptance. This is a bounded
+comparison policy, not a determination that the schema definitions conflict. `FULL` requires two
+bounded directions, and transitive strategies require bounded comparisons for every selected version.
+Total work still scales with the selected history. Sparse field numbers will not incur work
+proportional to the largest tag. No network lookup or application class loading will be required.
+
+## Public-facing Changes
+
+### Public API
+
+No client, REST, or checker interface method signatures will change. Selecting the advanced checker
+will introduce stricter acceptance through existing incompatibility responses. Applications that
+rely on the root-name-only check may therefore encounter registration or consumer compatibility
+failures after activation.
+
+A comparison diagnostic will identify the requested strategy, the actual comparison direction, a
+rule, the message or field path and number where applicable, and concise writer/reader descriptions.
+The registry will attach the actual conflicting stored version, including when the failure comes
+from a native component of a `KEY_VALUE` schema. The checker will also report a SHA-256 fingerprint
+of the compared native schema bytes, which remains useful when it is invoked without registry
+metadata. A native component's fingerprint identifies that component; the version identifies the
+stored outer schema. Version numbers will not be inferred from collection order. For example:
+
+```text
+TYPE_CHANGED: existingSchemaVersion=3, strategy=BACKWARD, direction=BACKWARD,
+existingSchemaSha256=<64 hex digits>,
+path=example.Order.items.price, field=3, writer=INT64, reader=INT32
+```
+
+Other rule identifiers will include `ROOT_MESSAGE_CHANGED`, `FIELD_NUMBER_CHANGED`,
+`REQUIRED_FIELD_NOT_GUARANTEED`, `MAP_CHANGED`, `CARDINALITY_CHANGED`, `DEFAULT_CHANGED`,
+`UTF8_VALIDATION_ADDED`, `ONEOF_CONFLICT`, `ENUM_VALUE_NOT_READABLE`, `UNSUPPORTED_FEATURE`,
+and `COMPARISON_LIMIT_EXCEEDED`.
+
+Failures before descriptor comparison will have separate diagnostics. `ALWAYS_INCOMPATIBLE` and
+`UNKNOWN_STRATEGY` will identify strategy rejection. If reconstruction fails and readable metadata
+identifies an unsupported language or feature, the result will be `UNSUPPORTED_FEATURE`; other
+reconstruction failures will produce `SCHEMA_RECONSTRUCTION_FAILED`. These failures will not include
+a comparison direction, field path, or schema fingerprint. If such a failure is associated with a
+selected historical entry, the registry will still identify that entry's version. A failure to
+reconstruct the proposed schema will not be attributed to an unrelated historical version.
+
+Comparison diagnostics will report the first failure and be limited to 2,048 characters. Error text
+will remain human-readable rather than becoming a new machine-readable response format. Complete
+descriptors and message payloads will not be included.
+
+### Configuration
+
+The default value of `schemaRegistryCompatibilityCheckers` will remain unchanged. To opt in, replace:
+
+```text
+org.apache.pulsar.broker.service.schema.ProtobufNativeSchemaCompatibilityCheck
+```
+
+with:
+
+```text
+org.apache.pulsar.broker.service.schema.ProtobufNativeSchemaAdvancedCompatibilityCheck
+```
+
+For the standard checker list, the resulting configuration will be:
+
+```properties
+schemaRegistryCompatibilityCheckers=org.apache.pulsar.broker.service.schema.JsonSchemaCompatibilityCheck,org.apache.pulsar.broker.service.schema.AvroSchemaCompatibilityCheck,org.apache.pulsar.broker.service.schema.ProtobufNativeSchemaAdvancedCompatibilityCheck,org.apache.pulsar.broker.service.schema.ExternalSchemaCompatibilityCheck
+```
+
+Deployment-specific checker entries should be preserved. Exactly one checker will be selected for
+`PROTOBUF_NATIVE`; checkers for the same schema type will not form a validation chain.
+
+Broker startup will fail whenever a configured checker cannot be instantiated, including a
+misspelled class name, whether or not the advanced checker is selected. It must not silently replace
+schema enforcement with an empty registry after a checker-loading error. Other initialization behavior
+for configurations without the advanced checker will remain unchanged.
+
+When the advanced checker is selected, startup will also fail if another checker handles
+`PROTOBUF_NATIVE`, including the default or a custom checker, or if schema storage is unavailable or
+schema registry initialization fails. Both `broker.conf` and the generated configuration reference
+will describe selection, restart and reconnection effects, and the required rollback order.
+
+This will be a broker-wide choice through an existing startup setting. No new dynamic switch or
+per-topic checker setting is proposed. Topic, namespace, and broker compatibility strategies will
+continue to control direction and history scope. Changing the checker will require restarting
+brokers with a consistent selection across all brokers that can own the affected topics.
+
+# Monitoring
+
+Operators should monitor schema incompatibility results, schema registry request latency, and
+producer or consumer connection failures during activation. Compare these with the deployment's
+baseline and use the returned rule and path to investigate rejected changes. Unsupported constructs
+should be distinguished from incompatible evolution.
+
+Existing schema registry metrics will continue to provide operation counts and request duration.
+No new metrics or labels are proposed. Schema names, field paths, and fingerprints will not become
+metric labels; the diagnostic will provide that detail.
+
+# Security Considerations
+
+The checker will operate on schema data supplied through existing authenticated and authorized paths.
+It will add no endpoint, privilege, network lookup, or cross-tenant history access. In particular,
+`Any` type URLs will not be fetched, and application classes will not be loaded to inspect a schema.
+
+Descriptors will remain untrusted input subject to existing schema validation and request limits.
+Comparison will enforce the per-version, per-direction work budget described above, including on
+recursive graphs and repeated enum references. Completed pairs will remain memoized throughout that
+comparison. Sparse field numbers must not cause resource use proportional to the maximum tag.
+Diagnostics will be bounded and contain only information from schemas involved in the authorized
+operation. Complete schema contents and message payloads must not be logged.
+
+# Backward & Forward Compatibility
+
+The default checker selection and its comparison of resolved root names will remain unchanged, as
+will schema storage, message encoding, and the Pulsar binary protocol. No schema-data migration will
+be required. Shared descriptor reconstruction will affect all native-schema callers: it will detect
+invalid imports explicitly, resolve nested roots without a package, and preserve Java feature options.
+For non-canonical root names in packaged files, successful legacy lookup will take precedence over
+relative lookup, so an existing alias will continue to select the same message. Unqualified root
+names will remain readable. This applies to default-checker comparisons, upload and producer-registration
+validation, and client generic readers. Java feature preservation can change effective decoding behavior
+in updated client libraries, so representative schemas and messages should be exercised before an
+ordinary binary upgrade too.
+
+A configured checker that cannot be instantiated will now prevent broker startup for all checker
+selections. This intentionally changes the previous fallback behavior for invalid configurations.
+
+Selecting the advanced checker will tighten evolution acceptance. Activation will not perform an
+eager scan or rewrite of stored history, but it requires broker restarts that disconnect clients.
+On reconnection, schema-bearing consumers other than `AUTO_CONSUME` will undergo the configured
+compatibility check again. Identical pairs will be reused; other selected historical versions will
+still be checked and can reject a consumer whose schema has not changed. First-registration and
+registration-reuse shortcuts do not certify the full history or exempt that consumer from checking it.
+
+## Upgrade
+
+1. Prepare an isolated staging cluster with the proposed broker and client versions. Exercise
+   representative stored native descriptors and message decoding, including Java feature settings and
+   legacy root names, before upgrading production binaries. Retaining the default checker does not
+   isolate applications from shared deserializer changes.
+2. For advanced-checker activation, copy the active retained schema history of each representative
+   topic to staging topics in version order. Enable the advanced checker there; use
+   `ALWAYS_COMPATIBLE` while importing previously accepted history, then restore the intended
+   production strategy. Import still performs schema-data validation. Run compatibility requests
+   for prospective producer schemas and connect representative schema-bearing consumers with their
+   actual strategies. The `testCompatibility` endpoint uses the configured checker and registration
+   direction; it cannot by itself stand in for the consumer strategy mapping.
+3. Review failures against every version selected by transitive strategies. Include proto2 messages
+   with extension ranges, MessageSet, Java enum-behavior mismatches, and unregistered language feature
+   extensions. Even an unrelated field addition to an extension-bearing message will be unsupported.
+   Byte-identical reuse does not establish support for its future evolution. Consumer
+   `FULL_TRANSITIVE` can also reject a schema whose reading direction alone passes. Resolve these
+   cases or plan application/topic migrations; adding a new version does not repair earlier history.
+4. Upgrade production broker binaries while retaining the default native checker. Upgrading alone
+   will not enable field-level evolution checks. Correct invalid checker class settings before restart.
+5. Coordinate the checker selection across every broker that can own the topics. During rolling
+   activation, disable automatic schema updates through the namespace `isAllowAutoUpdateSchema`
+   policy (`pulsar-admin namespaces set-is-allow-auto-update-schema --disable tenant/namespace`),
+   or the broker `isAllowAutoUpdateSchemaEnabled` default where no namespace override applies.
+   Coordinate admin uploads and system-topic updates separately; neither is blocked by this policy.
+   Replicator auto-registration has its own allowance and must also be coordinated across clusters.
+   Do not use `ALWAYS_INCOMPATIBLE` as a registration freeze: it can reject consumers too.
+6. Replace the checker entry and restart brokers with a consistent configuration. Expect client
+   reconnections and possible rejection of unchanged consumer schemas against other retained
+   versions. Monitor connection failures, incompatibility diagnostics and registry latency.
+7. Restore the intended schema auto-update policies after activation is consistent and the
+   representative reconnections succeed.
+
+## Downgrade / Rollback
+
+Replace the advanced entry with `ProtobufNativeSchemaCompatibilityCheck` and restart brokers with a
+consistent configuration before starting any older binary that lacks the advanced class. Verify the
+configuration on every broker before downgrading binaries. Stored schemas will not need to be rewritten.
+
+This order is mandatory. An older binary started with the advanced class still configured may log a
+warning and fall back to `DefaultSchemaRegistryService`, which stores and returns no schemas and
+bypasses schema compatibility enforcement broker-wide. The new startup failure for invalid checker
+classes cannot protect an older binary. Misspelled checker names on the new binary will instead
+prevent startup.
+
+Rolling back the checker will restore the weaker acceptance policy. It will not undo registrations
+or repair application compatibility. A binary downgrade must separately account for whether the older
+Protobuf runtime can reconstruct every schema in use, especially Editions and their features. An
+unchanged storage format does not guarantee support for newer schema-language features.
+
+## Pulsar Geo-Replication Upgrade & Downgrade/Rollback Considerations
+
+Coordinate broker versions, checker selection, and compatibility strategies across clusters that
+share replicated topics. Review each cluster's retained schema history, since histories can differ.
+A schema accepted in one cluster can be rejected by an operation invoking the advanced checker in
+another cluster with different history or configuration.
+
+Complete binary upgrades before activating the advanced checker across participating clusters.
+Control schema evolution during the transition and monitor replication, schema registration, and
+consumer reconnection outcomes. Rollback should restore the default checker consistently before
+any binary downgrade, subject to the schema-language/runtime constraints above. No replication
+protocol change is proposed.
+
+# Alternatives
+
+1. **Keep the root-name-only check.** It preserves existing acceptance behavior but cannot identify
+   incompatible field changes. It will remain available for existing deployments.
+2. **Convert Protobuf schemas to Avro.** This introduces a second schema model and does not naturally
+   preserve Protobuf field numbers, `oneof`, and Editions features. Native descriptors already carry
+   the information needed for this proposal.
+3. **Compare custom field models or raw descriptor equality.** A separate model duplicates Protobuf
+   semantics; equality rejects harmless differences and cannot express directional reading.
+4. **Accept all conditionally wire-compatible conversions.** Their safety depends on application
+   values or upgrade order that the registry cannot establish. This proposal uses conservative rules.
+5. **Add another registry or schema linter as a broker dependency.** Its compatibility scope and
+   integration model may differ from the directional root-message contract. No such dependency is
+   needed to compare native descriptors.
+6. **Replace the default checker.** This would tighten acceptance for existing deployments without
+   an explicit activation step. Any default change requires separate migration planning and review.
+
+# General Notes
+
+Operator documentation will describe checker selection, the compatibility contract, supported-feature
+limits, diagnostics, and coordinated activation and rollback. It belongs in the Pulsar website
+repository. Support for additional Editions or relaxed compatibility rules should be proposed
+explicitly rather than inferred from a Protobuf dependency upgrade.
+
+# Links
+
+- Mailing List discussion thread: https://lists.apache.org/thread/gs6j28zgk8r2d15fxdzkbc01xlf80zhq
+- Mailing List voting thread: https://lists.apache.org/thread/39zhzfjwmlmdtzzkzbvm19bw51y0rno4
+- Protobuf references: [proto2 evolution](https://protobuf.dev/programming-guides/proto2/#updating),
+  [proto3 evolution](https://protobuf.dev/programming-guides/proto3/#updating),
+  [enum behavior](https://protobuf.dev/programming-guides/enum/),
+  [field presence](https://protobuf.dev/programming-guides/field_presence/), and
+  [Editions features](https://protobuf.dev/editions/features/).