[PR] [improve][build] Restore fastutil and minimize it in the client and server distributions [pulsar]

["lhotari (via GitHub)" <gi...@apache.org>]

lhotari opened a new pull request, #26032:
URL: https://github.com/apache/pulsar/pull/26032

   Closes #24996
   
   Reverts #25413 and addresses the review feedback in #26028.
   
   ### Motivation
   
   #25413 removed the `fastutil` dependency and replaced it with hand-written primitive
   collections under `org.apache.pulsar.common.util.collections` (`Int2ObjectOpenHashMap`,
   `IntIntPair`, `Long2ObjectOpenHashMap`, and similar). That change was driven by a real
   problem — the full `fastutil` jar is ~24 MB / ~12,965 classes and we don't want to ship
   all of it — but the conversion of the Maven build's fastutil *minification* to Gradle was
   the actual blocker, not fastutil itself.
   
   As discussed in the #26028 review, hand-rolled primitive collections are the wrong
   trade-off: custom collection implementations have repeatedly been a source of bugs and are
   an ongoing maintenance burden, and there is a large combinatorial space of primitive
   collection types we'd have to reimplement and maintain. fastutil is a mature library whose
   collections have been optimized for performance for many years, and when a new use case
   needs another primitive collection, fastutil already has it — it can be used directly
   without adding more custom classes to the Pulsar code base.
   
   Memory efficiency is a concrete reason this matters. The Key_Shared draining-hashes design
   in [PIP-379](https://github.com/apache/pulsar/blob/master/pip/pip-379.md) uses fastutil
   primitive collections specifically to keep the per-subscription tracking footprint small,
   and its memory model is computed directly on those types:
   - The draining-hashes tracker is a fastutil `Int2ObjectOpenHashMap<DrainingHashEntry>`,
     using primitive `int` keys to avoid boxed `Integer` keys.
   - The per-ledger pending-acks map is a `Long2ObjectSortedMap<IntIntPair>` backed by a
     `Long2ObjectRBTreeMap`, with `IntIntPair.of(batchSize, stickyKeyHash)` as the value —
     primitive `long` keys and a primitive int-pair value instead of boxed wrappers.
   - PIP-379's sizing (≈52 bytes/entry, padded to 80 to account for memory alignment and the
     `Int2ObjectOpenHashMap` overhead; bounded worst case, ~0 once hashes drain) is derived
     from these fastutil maps.
   
   Reimplementing these collections by hand undercuts both the correctness and the memory
   characteristics that design relies on, so there is a memory-usage regression unless
   fastutil is restored.
   
   Finally, this resolves #24996: the full ~24 MB fastutil inflates deployments that use the
   non-shaded `-original` clients (e.g. exceeding AWS Lambda's unpacked-size limit). With this
   change, consumers of `pulsar-client-original` get a ~1.3 MB minimized fastutil jar instead
   of the full one.
   
   ### Modifications
   
   - **Restore fastutil (revert of #25413).** Switch the broker and client back to fastutil
     in `NegativeAcksTracker`, `PendingAcksMap`, `Consumer`, `DrainingHashesTracker`,
     `InMemoryRedeliveryTracker`, `InMemoryDelayedDeliveryTracker`, and
     `PersistentStickyKeyDispatcherMultipleConsumers` (and their tests), preferring fastutil
     primitive sorted maps (`Long2ObjectRBTreeMap` / `Long2ObjectAVLTreeMap`) over
     `java.util.TreeMap`. Post-#25413 improvements are kept (Roaring bitmap usage, the
     delayed-delivery race fixes, and the `PendingAcksMap` O(1) `size` optimization from
     #26019) — only the collection types are swapped back. Delete the hand-written
     `org.apache.pulsar.common.util.collections.*` classes and tests. Re-add `fastutil`
     (`8.5.18`) to `pulsar-broker`, `pulsar-client`, and the version catalog.
   
   - **Reusable `pulsar.minimized-dependencies-conventions` plugin.** A library-agnostic
     build-logic convention (+ `MinimizedDependenciesExtension`, exposed as `minimizedJar { }`).
     A packaging module applies it, declares reachability roots as `api(project(...))`
     dependencies, and sets `minimizedDependencies` (`"group:name"` list) + `maxRetainedClasses`.
     It builds a Shadow jar bundling only the configured libraries, runs `minimize()` (seeded
     from the `api` roots, since Shadow's `UnusedTracker` uses the project's own + `api`-scoped
     classes and these modules have no source), strips the build-only `api` seed from the
     `apiElements`/`runtimeElements` variants, and adds a config-cache-compatible
     `verifyMinimizedJar` check (wired into `check`) that fails if the retained class count
     exceeds the limit.
   
   - **Two minimized modules.** `pulsar-client-fastutil-minimized` (roots: `pulsar-client-original`,
     `pulsar-client-tools`, `pulsar-client-admin-original`; ~591 classes) and
     `pulsar-broker-fastutil-minimized` (roots: `pulsar-broker` + `pulsar-client-original`,
     ~818 classes — a superset of the client set).
   
   - **Client shaded jars** (`pulsar-client-shaded`, `-all`, `-admin-shaded`) exclude
     `it.unimi.dsi:fastutil` from `pulsar-client(-admin)-original` and bundle
     `pulsar-client-fastutil-minimized`; the client shade conventions relocate
     `it.unimi.dsi.fastutil` under the shade prefix.
   
   - **Server and shell distributions** exclude the full fastutil jar from their `distLib` and
     ship a minimized one instead: the server bundles the (unrelocated)
     `pulsar-broker-fastutil-minimized`, and the pulsar-shell CLI bundles
     `pulsar-client-fastutil-minimized`. Both binary LICENSE files carry a free-text fastutil
     (Apache-2.0) attribution without a jar filename — accepted by `checkBinaryLicense`, since
     the classes now ship inside a Pulsar-owned jar.
   
   - **`pulsar-client-original` published metadata** (publication-only) rewrites the POM
     (`pom.withXml`) and Gradle Module Metadata (post-processing the generated `.module` JSON)
     to replace the `it.unimi.dsi:fastutil` dependency with
     `org.apache.pulsar:pulsar-client-fastutil-minimized`. There is no build-graph dependency
     on the minimized module — intra-build, `pulsar-client-original` still exposes full fastutil
     (the broker uses more classes than the client set). `pulsar-client-fastutil-minimized` is
     published; the broker variant is not.
   
   - **Version bumps.** fastutil `8.5.18`; GradleUp Shadow `9.4.1` → `9.4.2`.
   
   Resulting sizes (built and measured):
   
   | Jar | Size | Classes |
   |-----|------|---------|
   | `pulsar-client-fastutil-minimized` | ~1.3 MB | 591 |
   | `pulsar-broker-fastutil-minimized` | ~1.7 MB | 818 |
   | Full `fastutil-8.5.18.jar` | ~24 MB | 12,965 |
   
   ≈17× smaller (client) and ≈14× smaller (broker), keeping exactly the fastutil classes the
   broker and client actually reach.
   
   ### Verifying this change
   
   This change is already covered by existing broker and client unit and integration tests —
   the collection types are swapped back to fastutil while behavior (and the kept post-#25413
   improvements) is unchanged, so the Key_Shared, negative-ack, redelivery, delayed-delivery,
   and pending-ack suites exercise it. The build-logic additions are guarded by the
   `verifyMinimizedJar` check and by `checkBinaryLicense` for the distribution LICENSE changes.
   CI passes (build, unit, and integration suites, including the shaded-jar and distribution
   assembly steps).
   
   ### Does this pull request potentially affect one of the following parts:
   
   - [x] Dependencies (add or upgrade a dependency)
   - [ ] The public API
   - [ ] The schema
   - [ ] The default values of configurations
   - [ ] The threading model
   - [ ] The binary protocol
   - [ ] The REST endpoints
   - [ ] The admin CLI options
   - [ ] The metrics
   - [x] Anything that affects deployment
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [improve][build] Restore fastutil and minimize it in the client and server distributions [pulsar]

["merlimat (via GitHub)" <gi...@apache.org>]

merlimat merged PR #26032:
URL: https://github.com/apache/pulsar/pull/26032


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org