You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "merlimat (via GitHub)" <gi...@apache.org> on 2026/04/14 18:14:48 UTC
[PR] [improve][build] Upgrade Netty from 4.1.132.Final to 4.2.12.Final [pulsar]
merlimat opened a new pull request, #25522:
URL: https://github.com/apache/pulsar/pull/25522
### Motivation
This upgrade brings Pulsar onto the Netty 4.2 line. The direct motivation is to unblock the async-http-client 3.x upgrade (#25023), which transitively depends on Netty 4.2 and cannot be landed while Pulsar force-pins Netty 4.1 — AHC 3.x bytecode is compiled against Netty 4.2 and would silently be downgraded to Pulsar's Netty 4.1 at resolution time, with `NoSuchMethodError` / `NoClassDefFoundError` surfacing in integration tests (which is what #25023's CI is hitting).
Netty 4.1 and 4.2 cannot co-exist on the classpath (same `io.netty.*` package namespace), so the upgrade has to be done in a single step. The Netty team asserts source/binary forward compatibility from 4.1 to 4.2 for regular API users:
- Announcement: https://netty.io/news/2025/04/03/4-2-0.html
- Migration guide: https://github.com/netty/netty/wiki/Netty-4.2-Migration-Guide
### Modifications
**`gradle/libs.versions.toml`**
- Bump `netty` from `4.1.132.Final` to `4.2.12.Final`.
- Drop the separate `netty-iouring = "0.0.26.Final"` version. io_uring has graduated from incubator (`io.netty.incubator:netty-incubator-transport-*-io_uring`) to a first-class Netty artifact (`io.netty:netty-transport-{classes,native}-io_uring`) and is now pinned to the same Netty version.
**`pulsar-common/build.gradle.kts`**
- Point the io_uring consumer at the renamed aliases.
**`pulsar-common/.../EventLoopUtil.java`**
- Netty 4.2 removed the dedicated `IOUringEventLoopGroup` class. io_uring now uses the generic `MultiThreadIoEventLoopGroup` + `IoUringIoHandler.newFactory()` pattern, which makes io_uring groups indistinguishable from any other `MultiThreadIoEventLoopGroup` by type — breaking the existing `instanceof`-based channel class dispatch in `getClientSocketChannelClass`, `getServerSocketChannelClass`, `getDatagramChannelClass`. Fix: introduce a private marker subclass `IoUringMultiThreadIoEventLoopGroup` used at construction so `instanceof` keeps discriminating.
- Repoint imports from `io.netty.incubator.channel.uring.*` to `io.netty.channel.uring.*` and adjust class names (`IOUring` → `IoUring`).
**`build-logic/conventions/.../pulsar.java-conventions.gradle.kts`**
- Exclude `io.netty.incubator` from all configurations. BookKeeper 4.17.3 (`bookkeeper-common` and `stream-storage-java-client`) still declares a transitive dependency on the `0.0.26.Final` incubator io_uring jars. Those jars are compiled against Netty 4.1 internals and are not safe to leave on the 4.2 classpath. Pulsar uses the core io_uring API via `EventLoopUtil`; BK stream-storage is an optional feature that Pulsar does not expose in its default surface.
**`distribution/{server,shell}/src/assemble/LICENSE.bin.txt`**
- Bump all `4.1.132.Final` entries to `4.2.12.Final`.
- Replace the monolithic `netty-codec-*.jar` with its 4.2 split-out sub-modules `netty-codec-base` and `netty-codec-compression` (`netty-codec` is now an aggregator POM that ships no classes).
- Rename the incubator io_uring entries to the core io_uring artifacts.
- The jar set was cross-checked against the output of `:distribution:pulsar-server-distribution:serverDistTar` and `:distribution:pulsar-shell-distribution:shellDistTar`.
**`pulsar-common/.../BitSetRecyclableRecyclableTest` and `ConcurrentBitSetRecyclableTest`**
- Relax the `testRecycle` assertion. Netty 4.2's `io.netty.util.Recycler` (itself deprecated in 4.2) no longer guarantees same-thread immediate reuse, so we only assert functional behavior: any recycled instance must come back cleared, and distinct `create()` calls must return distinct objects.
### Verification
- `./gradlew compileJava compileTestJava` — clean across the entire project. Only deprecation warnings for the 4.2 compat shims (`NioEventLoopGroup`, `EpollEventLoopGroup`, `DefaultEventLoopGroup`, `ChannelOption.RCVBUF_ALLOCATOR`, `EpollMode`, `Recycler`, `PlatformDependent.threadLocalRandom`). These are still functional in 4.2; cleanup can follow in a separate PR.
- `:pulsar-common:test` — passes.
- `:pulsar-broker:test --tests BrokerServiceTest` — passes (broker startup, producer/consumer flow, Netty transport end-to-end).
- `:pulsar-proxy:test --tests ProxyServiceTlsStarterTest` — passes (proxy, TLS handshake, tcnative-boringssl integration).
- `:distribution:pulsar-server-distribution:serverDistTar` and `:distribution:pulsar-shell-distribution:shellDistTar` — both build and the Netty jar set inside each tarball matches `LICENSE.bin.txt` exactly.
### Out of scope (follow-ups)
- **TLS endpoint identification default change.** Netty 4.2 changes `SslContextBuilder.endpointIdentificationAlgorithm` default from `null` to `"HTTPS"`. Pulsar's TLS client sites need to be audited and explicitly configured. This is deliberately not in this PR because the audit touches many modules (`pulsar-client`, `pulsar-broker`, `pulsar-proxy`, `pulsar-broker-auth-oidc`, admin) and should land as its own reviewable change.
- **Default allocator change.** Netty 4.2 changes the default `ByteBufAllocator` from pooled to adaptive. Pulsar is insulated for its main allocator path because `PulsarByteBufAllocator` explicitly constructs a `PooledByteBufAllocator.DEFAULT` via BK's `ByteBufAllocatorBuilder` — so Pulsar's own allocation does not go through Netty's `ByteBufAllocator.DEFAULT`. If CI soak tests surface regressions in Netty-internal allocation paths, `-Dio.netty.allocator.type=pooled` can be added to the launch scripts as a follow-up.
- **Deprecation cleanups.** The 4.2 compat shims (`NioEventLoopGroup`, `EpollEventLoopGroup`, `RCVBUF_ALLOCATOR`, etc.) can be migrated to the new APIs in a follow-up. They all still work in 4.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][build] Upgrade Netty from 4.1.132.Final to 4.2.12.Final [pulsar]
Posted by "geniusjoe (via GitHub)" <gi...@apache.org>.
geniusjoe commented on PR #25522:
URL: https://github.com/apache/pulsar/pull/25522#issuecomment-4301505659
Based on the scenarios I’m currently using, when brokers and consumer networks are throttled, message dispatch can slow down, causing back pressure on the broker side and leading to high direct memory usage. If, at this time, the Pulsar ledger handler attempts to read entries from BK, it may encounter transient direct memory OOM due to direct memory exhaustion. However, in the current architecture, even with the `ExitOnOutOfMemoryError` JVM parameter configured, the system does not shut down. Instead, it backs off and retries the read, and the part of the entry bytebuf that was not successfully read is automatically reclaimed by Netty’s internal exception handling, preventing any memory leaks.
I personally think the current exception handling logic is quite good, as it helps avoid OOM that could be triggered by back pressure when brokers and consumer networks are throttled. I’m not entirely sure whether Netty 4.2’s `adaptiveByteBuf` is fully compatible with the logic of Netty 4.1’s `pooledByteBuf` in this part. When upgrading to a major version of Netty, if the new `adaptiveByteBuf` does not provide significant improvements in performance or other aspects, it may be worth evaluating whether to adopt the new adaptiveByteBufor continue reusing the old `pooledByteBuf` mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][build] Upgrade Netty from 4.1.132.Final to 4.2.12.Final [pulsar]
Posted by "geniusjoe (via GitHub)" <gi...@apache.org>.
geniusjoe commented on PR #25522:
URL: https://github.com/apache/pulsar/pull/25522#issuecomment-4305285845
> @geniusjoe The back pressure for BK reads is handled by configuring `managedLedgerMaxReadsInFlightSizeInMB `. Please see https://github.com/apache/pulsar/blob/master/pip/pip-442.md for more details about broker memory management and related backpressure configuration. The memory limits aren't accurate since a Netty ByteBuf can be holding on to a larger underlying buffer where it was split off.
>
> In tuning, it's necessary to set the limits low enough that memory limit doesn't ever reach the scenario where Netty direct memory would run out. That might be impossible to achieve in all cases. One of the reasons is what you touched upon in #25274. Memory allocations, either pooled or "direct" (handled by malloc/OS), will cause fragmentation over time and there will be less memory available in certain workloads.
>
>
> As mentioned in my comment [#25021 (comment)](https://github.com/apache/pulsar/issues/25021#issuecomment-3584774110), AutoMQ's blog post ["Challenges of Custom Cache Implementation in Netty-Based Streaming Systems: Memory Fragmentation and OOM Issues"](https://www.automq.com/blog/netty-based-streaming-systems-memory-fragmentation-and-oom-issues) describes some of the problems with fragmentation in the Netty PooledByteBufAllocator. The AutoMQ article doesn't evaluate AdaptiveByteBufAllocator. I'd assume that it handles caching usecases in a better way than PooledByteBufAllocator. It would be great to get some feedback from real tests.
Lari, thank you for your response. I think the `managedLedgerMaxReadsInFlightSizeInMB` configuration you mentioned is a good workaround to avoid back pressure in bandwidth‑limited scenarios.
I also agree with your point. Ideally, we could compare the two memory allocation algorithms—`AdaptiveByteBufAllocator` and `PooledByteBufAllocator` —under the same Pulsar conditions. By using the `top` command to observe the actual memory consumption of the Java process and leveraging the JVM’s Native Memory Tracking to obtain metrics on direct physical memory usage, we could likely reveal the performance differences between the two allocators.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][build] Upgrade Netty from 4.1.132.Final to 4.2.12.Final [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #25522:
URL: https://github.com/apache/pulsar/pull/25522#issuecomment-4255802822
FYI, Issue for AsyncHttpClient 2.12.x backport of GHSA-cmxv-58fp-fm3g: https://github.com/AsyncHttpClient/async-http-client/issues/2167
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][build] Upgrade Netty from 4.1.132.Final to 4.2.12.Final [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #25522:
URL: https://github.com/apache/pulsar/pull/25522#issuecomment-4302468787
> Based on the scenarios I’m currently using, when brokers and consumer networks are throttled, message dispatch can slow down, causing back pressure on the broker side and leading to high direct memory usage. If, at this time, the Pulsar ledger handler attempts to read entries from BK, it may encounter transient direct memory OOM due to direct memory exhaustion.
@geniusjoe The back pressure for BK reads is handled by configuring `managedLedgerMaxReadsInFlightSizeInMB `. Please see https://github.com/apache/pulsar/blob/master/pip/pip-442.md for more details about broker memory management and related backpressure configuration. The memory limits aren't accurate since a Netty ByteBuf can be holding on to a larger underlying buffer where it was split off.
In tuning, it's necessary to set the limits low enough that memory limit doesn't ever reach the scenario where Netty direct memory would run out. That might be impossible to achieve in all cases. One of the reasons is what you touched upon in #25274. Memory allocations, either pooled or "direct" (handled by malloc/OS), will cause fragmentation over time and there will be less memory available in certain workloads.
> I personally think the current exception handling logic is quite good, as it helps avoid OOM that could be triggered by back pressure when brokers and consumer networks are throttled. I’m not entirely sure whether Netty 4.2’s `adaptiveByteBuf` is fully compatible with the logic of Netty 4.1’s `pooledByteBuf` in this part. When upgrading to a major version of Netty, if the new `adaptiveByteBuf` does not provide significant improvements in performance or other aspects, it may be worth evaluating whether to adopt the new `adaptiveByteBuf` or continue reusing the old `pooledByteBuf` mode.
As mentioned in my comment https://github.com/apache/pulsar/issues/25021#issuecomment-3584774110, AutoMQ's blog post ["Challenges of Custom Cache Implementation in Netty-Based Streaming Systems: Memory Fragmentation and OOM Issues"](https://www.automq.com/blog/netty-based-streaming-systems-memory-fragmentation-and-oom-issues) describes some of the problems with fragmentation in the Netty PooledByteBufAllocator. The AutoMQ article doesn't evaluate AdaptiveByteBufAllocator. I'd assume that it handles caching usecases in a better way than PooledByteBufAllocator. It would be great to get some feedback from real tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][build] Upgrade Netty from 4.1.132.Final to 4.2.12.Final [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on PR #25522:
URL: https://github.com/apache/pulsar/pull/25522#issuecomment-4308745975
Deferring this upgrade until BK 4.18, which will also be using Netty 4.2.x
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][build] Upgrade Netty from 4.1.132.Final to 4.2.12.Final [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat closed pull request #25522: [improve][build] Upgrade Netty from 4.1.132.Final to 4.2.12.Final
URL: https://github.com/apache/pulsar/pull/25522
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org