[DISCUSS] PIP-494: Scalable Topics Client Specification — authoritative reference and change process

[Matteo Merli <ma...@gmail.com>]

https://github.com/apache/pulsar/pull/26447
Rendered version:
https://github.com/merlimat/pulsar/blob/mmerli/pip-494/pip/pip-494.md

------

# PIP-494: Scalable Topics Client Specification — authoritative
reference and change process

*Sub-PIP of [PIP-460: Scalable Topics](pip-460.md)*

> **Draft.** This PIP establishes a **process**; it deliberately does not restate the specification's
> content, which lives in the repository at `spec/scalable-topics/`. The current work-in-progress
> specification —
> <https://github.com/merlimat/pulsar/tree/mmerli/scalable-topics-spec/spec/scalable-topics> — will be
> adopted as **version 1.0** of the specification. It reflects the current client API and
> semantics as designed in the scalable-topics PIPs ([PIP-460](pip-460.md), [PIP-468](pip-468.md),
> [PIP-483](pip-483.md), [PIP-486](pip-486.md)) and as implemented on the `master` branch.

## Motivation

Scalable topics ([PIP-460](pip-460.md) and its sub-PIPs — the
controller [PIP-468](pip-468.md), auto
split/merge [PIP-483](pip-483.md), key-shared consumption
[PIP-486](pip-486.md)) introduce a client model
that is materially different from classic Pulsar topics: no
application-visible partitions, a dynamic
per-cluster segment layout the client must track, client-side key
routing, three distinct consumer
modes with different ordering/acknowledgment contracts, a checkpoint
position type, a controller-driven
consumer-assignment protocol, and a family of new binary-protocol commands.

Today the only complete description of that client behavior is the
**Java V5 client's source code**, with
the design rationale scattered across the PIPs. That is not a workable
basis for the ecosystem:

- **Other-language SDKs** (Go, Python, C++, Rust, Node, .NET) need a
precise, language-neutral definition
  of the API contract, the required client-side mechanisms, and the
exact wire interactions — not a Java
  implementation to reverse-engineer.
- **Interoperability** depends on every client routing keys
identically, applying layouts and assignments
  with the same staleness rules, and honoring the same ordering and
acknowledgment semantics. Without a
  normative reference, divergent clients are inevitable and hard to diagnose.
- **PIPs are design proposals, not contracts.** They explain *why* and
record decisions, but they are
  point-in-time documents that are never updated after acceptance. The
living behavior needs a living,
  normative document.
- **Stability and deprecation** need defined rules so SDK authors know
what they can rely on across
  Pulsar releases.

This PIP establishes a **Scalable Topics Client Specification** as the
single authoritative description
of the client API, its semantics, and its protocol-level interactions
— the base document for anyone
implementing scalable-topics support in a Pulsar client SDK — and
defines the **process** by which that
specification is maintained and changed.

## Background knowledge

**PIPs vs. a specification.** A PIP proposes and ratifies a change;
once merged it is a historical
record of a decision. A specification is a *current* normative
statement of behavior that implementations
conform to, and it evolves as behavior evolves. Mature
cross-implementation ecosystems keep both: the
OpenTelemetry project, for example, maintains a versioned
specification with explicit stability tiers and
a formal change process, separate from the proposals that motivated
each change. This PIP adopts that
model for scalable topics.

**Reference implementation.** The Java V5 client
(`pulsar-client-api-v5`, `pulsar-client-v5`) is the
reference implementation of scalable-topics client behavior. The
specification is derived from it and
verified against it, but the specification — not the Java code — is
the source of truth for conformance.

## Goals

### In Scope

- Establish `spec/scalable-topics/` in the `apache/pulsar` repository
as the **authoritative,
  language-neutral specification** for scalable-topics clients: the
API contract, its semantics, the
  required client-side mechanisms, and the wire-protocol interactions.
- Define the specification as the **required base for any new client
SDK** adding scalable-topics
  support, and the reference against which SDK **conformance** is judged.
- Define the **change process**: how normative changes to the
specification are proposed, reviewed, and
  landed, and how the specification is kept in sync with the reference
implementation.
- Define the specification's **versioning, stability tiers, and
deprecation/removal rules**, aligned with
  the Apache Pulsar release (LTS) cadence.

### Out of Scope

- The specification's content itself. It is a separate deliverable
(already drafted) and is not
  restated here; this PIP governs it, it does not define client behavior.
- Client behavior for classic (non-scalable) topics — the
specification covers scalable topics and their
  migration bridge from classic topics only.
- The mechanics of publishing the specification to the user-facing
documentation site
  (`apache/pulsar-site`); that is a follow-up task once the process is in place.

## High Level Design

### What is established

1. **Location and status.** The specification lives at
`spec/scalable-topics/` in `apache/pulsar`, as a
   set of Markdown documents with a top-level index (`README.md`). It
is a first-class artifact of the
   project, maintained alongside the code it describes.

2. **Authority.** For scalable-topics client behavior, the
specification is **normative**. Where an
   implementation (including the Java reference client) and the
specification disagree, the
   specification is the source of truth and the implementation is the
bug — unless a PIP changes the
   specification.

3. **Audience and use.** The specification is the base document for
implementing a scalable-topics
   client in any language. It separates the transport-agnostic **API
contract** (what an application
   observes: operations, inputs, guarantees, errors) from the
**implementation requirements** (what a
   client must do internally: routing, layout tracking, fan-out,
retries) and from the **wire protocol**
   (the exact command exchanges, with sequence diagrams), so an
implementer can build against the
   contract and bind it to the Pulsar protocol.

4. **Conformance.** The specification defines what a conformant client
MUST implement (the core
   producer, the three consumer modes, transactions) and which
capabilities are optional. An SDK claims
   scalable-topics support by conforming to the specification's
conformance document.

### How it is governed

The specification is **not itself a PIP**. It is a living document
maintained under the change process
below, which ties every normative change to a PIP and keeps the
document in lockstep with the reference
implementation.

## Detailed Design

### 1. Change process

- **Every normative change to the specification MUST be made through a
PIP.** A "normative change" is
  any change to a requirement (MUST/SHOULD/MAY), a guarantee, an
operation's contract, or a wire
  interaction. The PIP carries the design and rationale; reviewers
evaluate the proposed behavior and the
  specification edits **together**.
- **Specification edits land with the PIP.** When a PIP that changes
scalable-topics client behavior is
  accepted, the corresponding edits to `spec/scalable-topics/` are
merged together with it — normally in
  the **same pull request** as the implementation — so the contract
and its decision record never
  drift. A PIP that changes client-observable behavior is not complete
until its specification edits are
  merged.
- **Non-normative edits** (typos, wording clarifications that change
no requirement, added examples or
  diagrams, cross-reference fixes) do not require a PIP and follow the
normal pull-request review.
- **PIPs remain the rationale and decision record**; the specification
is the normative contract kept
  continuously in sync with the ratified PIPs. A reader who wants to
know *what* a client must do reads
  the specification; a reader who wants to know *why* follows its
references to the PIPs.

### 2. Relationship to the reference implementation

- The Java V5 client is the **reference implementation**. The
specification's Stable content describes
  behavior that is **implemented in a released broker and the
reference client, or ratified via an
  accepted PIP and committed to ship**; an SDK built to it
interoperates with current and announced
  brokers.
- A change to the reference implementation that alters
client-observable behavior MUST be accompanied
  by the matching specification change (via §1). Conversely, the
specification MUST NOT describe
  behavior the reference implementation does not (or will not, per an
accepted PIP) exhibit.
- The specification includes a **Java reference mapping** appendix
that maps each language-neutral
  operation and type to the Java V5 surface, so the reference
implementation and the specification can
  be checked against each other and other SDKs can align their naming.

### 3. Versioning, aligned with Pulsar LTS releases

- The specification carries its own version of the form `MAJOR.MINOR`,
declared in its index.
- **MAJOR** increments on a backward-incompatible change to a *Stable*
requirement and is **tied to the
  Apache Pulsar LTS cadence**: each new MAJOR corresponds to a new
Pulsar LTS release. Backward-incompatible
  changes and removals of deprecated features therefore land only at
an LTS boundary, never in an interim
  release.
- **MINOR** increments on backward-compatible additions or
clarifications and may occur in any release.
- The work-in-progress draft is currently at `0.x` (**Draft**).
Acceptance of this PIP adopts it as
  **`1.0`**, from which point the stability guarantees below take full
effect. Any later MAJOR
  transition is a normative change made through the change process (§1).

### 4. Stability tiers and deprecation

Every feature in the specification carries a stability tier expressing
its change contract:

- **Stable** — will not change incompatibly without a MAJOR bump;
conformant clients MUST implement all
  Stable, non-optional requirements.
- **Experimental** — may change in any way, including removal, in a
MINOR version; clients MAY implement
  it, and omitting it never makes a client non-conformant. Used for
designs that are documentable but not
  yet settled.
- **Deprecated** — Stable but scheduled for removal; retained for
compatibility, not to be newly adopted.

**Deprecation and removal mechanism.** A feature enters Deprecated
only through the change process (a
PIP) that records the reason, the replacement (or that none exists),
and the earliest version at which
removal is permitted. While Deprecated, brokers MUST keep honoring it,
clients SHOULD migrate, and
implementations SHOULD emit a non-fatal deprecation signal. A
Deprecated Stable feature MUST NOT be
removed before the next MAJOR version **and** MUST remain available at
least until the next Pulsar LTS
release, guaranteeing an LTS-to-LTS upgrade path in which the feature
and its replacement coexist.

### 5. Scope of the specification

- **In scope:** everything a client SDK must implement for scalable
topics — topic identity, the data
  model (message, message identifier, checkpoint, schema), the
producer and the three consumer modes,
  transactions, error semantics, the client-side mechanisms, the
complete wire protocol, conformance
  criteria, and compatibility with classic topics (the migration bridge).
- **Out of scope:** broker-internal behavior (controller, split/merge
execution, storage) except where
  observable by a client; and designs whose specification is not yet
settled (currently geo-replication
  of scalable topics), which enter the specification — initially as
Experimental — only once ratified.

### 6. Obligations on new client SDKs

An SDK adding scalable-topics support:

- MUST implement the specification's core (Stable, non-optional)
requirements and MAY implement optional
  capabilities, each fully per its section, gating on broker support
where the specification requires.
- MUST use the specification — not the Java client's source — as its
reference, and SHOULD map its
  idiomatic API to the specification's language-neutral operations (as
the Java mapping appendix does for
  Java), so cross-SDK behavior stays comparable.
- SHOULD contribute clarifications back through the change process
when the specification is found
  ambiguous or incomplete, rather than resolving the ambiguity privately.

## Public-facing changes

- **New repository artifact:** `spec/scalable-topics/` — the
specification document set. The
  work-in-progress draft linked above is adopted as **version 1.0**:
it reflects the current client
  API and semantics as designed in the scalable-topics PIPs and as
implemented on `master`; the
  stability guarantees of §3–§4 apply from that version.
- **Process:** the change process in §1 applies to every subsequent
PIP that touches scalable-topics
  client behavior, and to pull requests changing the reference
implementation's client-observable
  behavior.
- No code, wire-protocol, configuration, API, or metrics changes are
introduced by this PIP.

## Backward & Forward Compatibility

Not applicable to this PIP itself: it introduces a document and a
process, not a behavior change. The
compatibility guarantees the specification *makes* (protocol
extensibility by addition, cross-version
restorability of serialized identifiers and checkpoints, the
LTS-anchored deprecation window) are defined
within the specification and are governed by §3–§4 of this PIP.

## Alternatives

- **Keep the PIPs as the only description.** Rejected: PIPs are
point-in-time proposals that are never
  updated after acceptance; they cannot serve as a living, normative
contract, and they describe design
  rationale rather than exact client obligations.
- **Make the specification itself a PIP.** Rejected: a PIP is ratified
once and then frozen, whereas the
  specification must evolve continuously with the implementation.
Governing the specification *through*
  PIPs (§1) captures the review rigor without freezing the document.
- **Per-language documentation only (e.g. Javadoc as the reference).**
Rejected: language-specific
  documentation cannot define language-neutral contracts, wire
interactions, or conformance, and it
  invites divergence between SDKs.
- **Publish the specification only on the documentation site, not in
the repository.** Rejected for the
  source of truth: keeping it in `apache/pulsar` lets specification
edits merge in the same pull request
  as the implementation change (§1). Mirroring to the docs site is a follow-up.

## Links

<!-- Updated afterwards -->
* Specification (work in progress):
  <https://github.com/merlimat/pulsar/tree/mmerli/scalable-topics-spec/spec/scalable-topics>
  — will live at `spec/scalable-topics/` in `apache/pulsar` (index: `README.md`)
* Related: [PIP-460](pip-460.md), [PIP-468](pip-468.md),
[PIP-483](pip-483.md), [PIP-486](pip-486.md)


--
Matteo Merli
<ma...@gmail.com>