You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "lhotari (via GitHub)" <gi...@apache.org> on 2026/09/04 19:20:09 UTC

[PR] [fix][cli] Apply PULSAR_MEM and PULSAR_GC in bin/pulsar-perf [pulsar]

lhotari opened a new pull request, #26465:
URL: https://github.com/apache/pulsar/pull/26465

   ### Motivation
   
   `bin/pulsar-perf` ignores `PULSAR_MEM` and `PULSAR_GC` entirely, so the tool always runs on the JVM's
   default heap ergonomics no matter what the caller exports. Running it in a container with
   
   ```
   PULSAR_MEM=-Xms512m -Xmx1g -XX:+UseTransparentHugePages -XX:+AlwaysPreTouch \
     -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/testoutput
   ```
   
   produces this command line — no heap setting, no collector setting, nothing from either variable:
   
   ```
   exec /opt/jvm/bin/java -cp /pulsar/conf:::/pulsar/lib/*: \
     -Dio.netty.recycler.maxCapacityPerThread=4096 -Dio.netty.allocator.maxOrder=10 \
     -XX:+ExitOnOutOfMemoryError -Dpulsar.allocator.exit_on_oom=true ... \
     org.apache.pulsar.testclient.PulsarPerfTestTool /pulsar/conf/client.conf produce test
   ```
   
   There are two conventions in the launcher scripts, and `pulsar-perf` falls between them:
   
   - `conf/pulsar_tools_env.sh` folds `PULSAR_MEM`, `PULSAR_GC` and `PULSAR_GC_LOG` into
     `PULSAR_EXTRA_OPTS` itself, so a script that sources it only has to append `$PULSAR_EXTRA_OPTS`.
     That is what `bin/pulsar-admin-common.sh` does, covering `pulsar-admin`, `pulsar-client` and
     `pulsar-shell`.
   - `conf/pulsar_env.sh` merely *defines* the variables, so a script that sources it has to append them
     explicitly. That is what `bin/pulsar` does at line 328.
   
   `bin/pulsar-perf` sources `conf/pulsar_env.sh` but appends only `$PULSAR_EXTRA_OPTS`, so the
   variables are set and then dropped. It is the only script with this mismatch: `bin/pulsar-daemon`
   delegates to `bin/pulsar`, `bin/function-localrunner` appends them itself, and the admin tools go
   through the tools env file.
   
   Noticed while working on #26460, where `PulsarProfilingTest` sets `PULSAR_MEM` and `PULSAR_GC` on its
   `pulsar-perf` containers and neither takes effect.
   
   ### Modifications
   
   Record whether the caller set `PULSAR_MEM` and `PULSAR_GC` before `conf/pulsar_env.sh` is sourced,
   and append each to the options only when they did.
   
   Taking the values after sourcing would change behaviour for everyone who sets neither, because
   `conf/pulsar_env.sh` fills them in with the broker's `-Xms2g -Xmx2g -XX:MaxDirectMemorySize=4g`. A
   load generator that suddenly reserved a 2g heap could fail to start where it used to run, on a small
   machine or a tightly capped container. Applying only what the caller passed keeps that case exactly
   as it was.
   
   Two consequences worth knowing:
   
   - Setting these by editing `conf/pulsar_env.sh` does not affect `pulsar-perf`; they have to be
     exported. An edited file is indistinguishable from the shipped default at the point the script
     reads it.
   - `PULSAR_GC_LOG` is deliberately left out, even though `bin/pulsar` includes it:
     `conf/pulsar_env.sh` always assigns it a value, so adding it here would make every `pulsar-perf`
     invocation start writing GC log files into the log directory.
   
   ### Verifying this change
   
   - [x] Make sure that the change passes the CI checks.
   
   This change is a launcher-script fix with no test coverage in the repository. It was verified by
   tracing the composed command line with `bash -x`:
   
   | caller sets | heap/GC flags on the exec line |
   |---|---|
   | nothing | none — unchanged from before this fix |
   | `PULSAR_MEM` only | exactly the value passed |
   | `PULSAR_GC` only | exactly the value passed |
   | both, as in the report above | `-Xms512m -Xmx1g -XX:+UseTransparentHugePages -XX:+AlwaysPreTouch -XX:+UseZGC -XX:+ZGenerational` |
   | `PULSAR_MEM=` (empty) | none, treated as unset |
   
   No `-Xlog:`/`-Xloggc:` argument is introduced in any case. `shellcheck -S warning` reports nothing on
   the changed lines. For contrast, `bin/pulsar broker` was traced with the same environment before the
   change and already passed the flags through, confirming the difference was in `pulsar-perf` alone.
   
   ### Does this pull request potentially affect one of the following parts:
   
   *If the box was checked, please highlight the changes*
   
   - [ ] Dependencies (add or upgrade a dependency)
   - [ ] The public API
   - [ ] The schema
   - [ ] The default values of configurations
   - [ ] The threading model
   - [ ] The binary protocol
   - [ ] The REST endpoints
   - [ ] The admin CLI options
   - [ ] The metrics
   - [ ] Anything that affects deployment
   
   `PULSAR_MEM` and `PULSAR_GC` now take effect for `pulsar-perf` where they previously did nothing, so
   anyone already exporting them will see the JVM settings they asked for. Callers who set neither are
   unaffected, which is why only caller-supplied values are applied.
   
   ### Documentation
   
   - [ ] `doc-required`
   - [x] `doc-not-needed`
   
   The variables are already documented; this makes `pulsar-perf` honour them.
   
   - [ ] `doc`
   - [ ] `doc-complete`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [fix][cli] Apply PULSAR_MEM and PULSAR_GC in bin/pulsar-perf [pulsar]

Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat merged PR #26465:
URL: https://github.com/apache/pulsar/pull/26465


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org