You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "merlimat (via GitHub)" <gi...@apache.org> on 2026/06/29 20:27:13 UTC

[PR] [improve][broker] PIP-486: entry-bucket wire format, hash & layout foundation [pulsar]

merlimat opened a new pull request, #26114:
URL: https://github.com/apache/pulsar/pull/26114

   ### Motivation
   
   This is the **first implementation PR for [PIP-486: Scalable Topic Key-Shared Consumption](https://github.com/apache/pulsar/blob/master/pip/pip-486.md)** (the PIP was merged in #26077).
   
   PIP-486 adds key-shared (per-message-key) ordered consumption on scalable topics via producer-side
   **entry-bucketing**, so multiple consumers can share one segment without coupling the producer's
   batching mode to the consumer's subscription mode. The full feature lands across a series of PRs; this
   one is the **protocol + hash + layout foundation** and is intentionally **behavior-neutral** — with a
   single entry-bucket per segment (the default), everything behaves exactly as today.
   
   ### Modifications
   
   - **Wire format (`MessageMetadata`):** add optional `entry_hash_min`/`entry_hash_max` (the effective
     entry-bucket hash range a batch covers; set by scalable-topic producers in a later PR).
   - **Bucket hash:** `Murmur3_32Hash.makeRawHash()` — the *unmasked* 32-bit hash. `makeHash()` clears the
     sign bit, which would confine the high 16 bits to `[0, 0x7FFF]`; the raw hash keeps both halves
     full-range. `SegmentRouter` now routes **segments on the high 16 bits** and exposes
     `entryBucketHash()` on the **low 16 bits** (independent ring, PIP-486), with `murmur()` so a caller
     hashes once and splits.
   - **Per-segment layout:** `SegmentInfoProto` carries `entry_bucket_splits` — an explicit, possibly
     **uneven** list of bucket split points (empty = a single bucket spanning the whole ring). Plumbed
     through `SegmentInfo`, `DagWatchSession`, and the V5 client (`ClientSegmentLayout` /
     `SegmentRouter.ActiveSegment`). Default is empty/`N=1`.
   
   All new fields default such that existing behavior is unchanged (`N=1`). Tests added for `makeRawHash`
   and the decoupled hash API; existing scalable tests updated for the new `SegmentInfo` shape.
   
   Subsequent PRs add producer entry-bucketing, broker bucket-routing + controller assignment, the V5
   consumer wiring, and the auto-scale policy.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org