[PR] [improve][test] Record Netty's allocator events in profiled performance runs, merge JFR configurations per component and profile on Alpine [pulsar]

["lhotari (via GitHub)" <gi...@apache.org>]

lhotari opened a new pull request, #26787:
URL: https://github.com/apache/pulsar/pull/26787

   ### Motivation
   
   The performance tests' profiled runs (`tests/performance`) record JDK Flight Recorder's events alongside async-profiler, but they didn't record anything about Netty's buffer allocators. Netty 4.2's `AdaptivePoolingAllocator` and `PooledByteBufAllocator` emit JFR events for their chunk and buffer allocations (`io.netty.*`, see Netty's `microbench/src/main/resources/Netty Allocator Events.jfc`). With them, a profile shows how much memory the allocators allocate and free, which buffers don't fit the pooled memory, and how often buffers grow. For example, in the IoT telemetry max-rate scenario the broker's allocator allocated and freed about 660 chunks per second, about 94 MB/s, at a steady load, which led to #26786.
   
   Recording those events brought up three more problems with the profiling setup:
   
   - **The JFR configuration was fixed.** It was the JDK's `profile` configuration, set with async-profiler's `jfrsync` option in each scenario's profiler options, and the configuration couldn't be extended with other events.
   - **Profiled runs used the glibc-based Wolfi test image.** The unprofiled runs and Pulsar's default Docker image are Alpine-based, and a profile of another image doesn't carry over: the libc's memory allocator, for one, behaves differently (#26786).
   - **On Alpine, stacks stopped at musl.** Alpine strips musl, so every frame in it read as `/lib/ld-musl-x86_64.so.1`, and the JNI frames above it were missing. That was 44 % of a broker's CPU samples in the max-rate scenario.
   
   A run whose applications received duplicate messages also counted as successful. The launcher then deleted `launcher.log`, which has the containers' logs, so a broker that ran out of direct memory and restarted mid-run left no trace of why.
   
   ### Modifications
   
   - **JFR configurations per component.** A profiled component's settings take `jfrConfigurations`, a list that the launcher merges with the JDK's `jfr configure`; the default is `[profile, pulsar.jfc]`.
     - A name, such as `profile`, is one of the JDK's configurations. A name ending with `.jfc` is a file of the new `tests/performance/jfr` directory.
     - Before the cluster starts, the launcher merges each component's list in a one-off container of the component's image, so that the JDK's configurations are those of the JVM that records with them.
     - The merged file is written as `jfr-configuration.jfc` beside the component's recordings and passed to async-profiler's `jfrsync` option. The profiler options no longer set `jfrsync`, and the launcher rejects options that do.
     - When the image's JDK can't merge them, such as a released Pulsar's image whose JDK has no `jfr` tool, the component records with the JDK's `profile` configuration, and the launcher says so.
   - **Netty's allocator events.**
     - `pulsar.jfc` adds `io.netty.AllocateChunk`, `FreeChunk`, `ReturnChunk` and `ReallocateBuffer`.
     - `netty-allocations.jfc` adds the events of every buffer allocation and free, `io.netty.AllocateBuffer` and `FreeBuffer`. These are costly: there is an event for every buffer, and JFR can't sample them, since its `throttle` setting doesn't apply to events without `@Throttle`. With them, a broker recording in the max-rate scenario grew from about 10 MB to 1 GB, and the broker didn't finish writing it before it was stopped.
     - `configs/profile-{broker,gateways,applications}-netty-allocations.yaml` profile a component with them, in place of `configs/profile-<component>`.
   - **Analytics.** After a profiled run, `NettyAllocatorEvents` (report tool) summarizes each measurement recording's Netty allocator events into `<recording>.measurement.netty-allocator.json`, and the profile report shows it in a "Netty allocator events" section:
     - the events per second and per message;
     - buffer and chunk allocations by allocator, memory and pooled or one-off chunk;
     - buffer sizes, reallocations, and the thread pools that allocate.
   
     The `summarizeNettyAllocatorEvents` task summarizes other recordings.
   - **Profiling on Alpine.**
     - Profiled runs use the Alpine test image by default, like the unprofiled runs; `-Pinttest.testImageVariant=wolfi` still profiles on the Wolfi image.
     - The Alpine test image installs musl's debug symbols, `musl-dbg`. apk pins it to the installed musl version, and it isn't installed on the Wolfi image. With it, a stack reads, for example, `Socket.recvAddress` → `netty_unix_socket_recvAddress` → `recvfrom` → `__syscall_cp_c`. Pulsar's own Docker image is unchanged.
   - **Keeping the logs of incorrect deliveries.** The launcher keeps `launcher.log` when the applications received duplicates, ordering violations or invalid messages.
   - **Documentation.** `docs/profiling.md` ("The JFR configuration"), `docs/analyzing-profiles.md` ("Netty allocator events"), `AGENTS.md`, `docs/running-scenarios.md`, `docs/run-reports.md`, `docs/comparing-revisions.md` and the scenario docs.
   
   ### Verifying this change
   
   - [ ] Make sure that the change passes the CI checks.
   
   This change added tests and can be verified as follows:
   
   - **Unit tests:**
     - `NettyAllocatorEventsTest` records stand-in events with Netty's event names and fields, and checks the summary and the report section.
     - `ProfilingSettingsTest` covers `jfrConfigurations`, including rejecting invalid entries and a `jfrsync` in the options.
     - `DeliveryCheckTest` covers keeping `launcher.log`.
   - **Profiled runs of the IoT telemetry max-rate scenario on one host:**
     - The broker and the gateways with the default configurations: recordings of 12.5 MB and 10.6 MB, and the profile reports' Netty sections.
     - The broker with the default configurations and the applications with `netty-allocations.jfc`, on the Alpine image: the merged `jfr-configuration.jfc` files, and `jfrsync` pointing at them in the agents' configurations.
     - The broker on the Alpine image with `musl-dbg`: CPU samples with an unnamed `ld-musl` frame went from 44.3 % (2,600 of 5,869) to none (0 of 5,798).
   
   ### Does this pull request potentially affect one of the following parts:
   
   <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
   
   *If the box was checked, please highlight the changes*
   
   - [ ] Dependencies (add or upgrade a dependency)
   - [ ] The public API
   - [ ] The schema
   - [ ] The default values of configurations
   - [ ] The threading model
   - [ ] The binary protocol
   - [ ] The REST endpoints
   - [ ] The admin CLI options
   - [ ] The metrics
   - [ ] Anything that affects deployment
   
   This change was prepared with the assistance of Claude Code (claude-opus-5-5); I have reviewed and verified it.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [improve][test] Record Netty's allocator events and other JFR settings per profiled component, and profile on Alpine [pulsar]

["lhotari (via GitHub)" <gi...@apache.org>]

lhotari merged PR #26787:
URL: https://github.com/apache/pulsar/pull/26787


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org