You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by lh...@apache.org on 2026/10/01 16:07:44 UTC
(pulsar) branch master updated: [improve][test] Add disk-limit options and A/B comparison charts to the Pulsar Performance Testing Framework (#26800)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new af8c5292febd [improve][test] Add disk-limit options and A/B comparison charts to the Pulsar Performance Testing Framework (#26800)
af8c5292febd is described below
commit af8c5292febdd54d8427d455482ecebeca1c8423
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Thu Oct 1 19:07:37 2026 +0300
[improve][test] Add disk-limit options and A/B comparison charts to the Pulsar Performance Testing Framework (#26800)
---
AGENTS.md | 2 +-
CONTRIBUTING.md | 34 +-
microbench/AGENTS.md | 2 +-
microbench/README.md | 8 +-
tests/README.md | 34 +-
.../docker-images/java-test-image/build.gradle.kts | 5 +
tests/integration/AGENTS.md | 4 +-
tests/performance/AGENTS.md | 348 +++++++++--------
tests/performance/README.md | 60 +--
tests/performance/docs/analyzing-profiles.md | 2 +-
tests/performance/docs/comparing-revisions.md | 40 +-
tests/performance/docs/metrics.md | 2 +-
tests/performance/docs/run-reports.md | 2 +-
tests/performance/environment/README.md | 96 +++--
.../scripts/configure-perf-test-environment.sh | 123 +++++-
.../performance/launcher/ClusterSettings.java | 88 ++++-
.../performance/launcher/PerformanceLauncher.java | 13 +-
.../performance/launcher/ClusterSettingsTest.java | 45 +++
tests/performance/report-tool/build.gradle.kts | 12 +
.../tests/performance/report/ComparisonCharts.java | 311 ++++++++++++++++
.../performance/report/ComparisonRenderer.java | 414 +++++++++++++++++++++
.../performance/report/ComparisonChartsTest.java | 206 ++++++++++
tests/performance/scenarios/README.md | 28 ++
.../scenarios/configs/bookie-journal-tmpfs.yaml | 25 ++
.../tools/src/main/dist/bin/run-workload | 3 +
25 files changed, 1647 insertions(+), 260 deletions(-)
diff --git a/AGENTS.md b/AGENTS.md
index 9767d0715594..ec1d414ce737 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -43,7 +43,7 @@ Some areas have an agent guide of their own, which routes to that area's docs; r
| Guide | Area |
|-------|------|
| [`tests/integration/AGENTS.md`](tests/integration/AGENTS.md) | The integration tests, which test Pulsar clusters in Docker containers. |
-| [`tests/performance/AGENTS.md`](tests/performance/AGENTS.md) | The performance tests, for running performance test experiments and automating them, including agent-based tuning. |
+| [`tests/performance/AGENTS.md`](tests/performance/AGENTS.md) | The Pulsar Performance Testing Framework, which runs a Pulsar cluster and its workloads in Docker on one host: performance experiments, profiling, A/B comparisons of revisions with charts, and automating them, including agent-based tuning. |
| [`microbench/AGENTS.md`](microbench/AGENTS.md) | The JMH microbenchmarks, for the performance of a single class or method. |
The authoritative project documentation is at <https://pulsar.apache.org>, whose source lives in the
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 85b14d111cc2..4e6468f30684 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -157,7 +157,7 @@ isolation, such as SASL's one class per worker, takes precedence.
Test JVMs write heap dumps on heap exhaustion to `/tmp/java_pid<PID>.hprof`, collected by CI's
existing failure artifacts. Set `-PtestHeapDumpPath=<existing-directory>` to use another directory.
-[The performance testing guide](tests/performance/README.md) leads to tools for analyzing them.
+[The Pulsar Performance Testing Framework](tests/performance/README.md) leads to tools for analyzing them.
Failed tests are retried once by default (`testRetryCount=1`; `0` when running inside the IDE). When
running tests locally, prefer **`-PtestRetryCount=0`** to catch failures (including flakiness) early
@@ -257,18 +257,19 @@ any integration test and without changing it. See
#### Profiling a performance scenario, including off-CPU time
-The performance tests' `profile` task profiles a whole scenario — a cluster and its workload
+The performance testing framework's `profile` task profiles a whole scenario — a cluster and its workload
applications — with three recorders running at the same time in each profiled JVM:
[async-profiler](https://github.com/async-profiler/async-profiler) samples CPU time and allocations,
JDK Flight Recorder (JFR) records the JVM's own events, such as monitor contention, thread parking and
garbage collection, into the same recording, and [jonoffcpu](https://github.com/jonoffcpu/jonoffcpu),
which bundles async-profiler, records from the kernel every interval in which a thread was blocked.
Joined to the Java stacks, those intervals give **off-CPU** profiles: where threads wait on locks,
-monitors, queues, I/O or GC, not only where they use CPU. See [Performance tests](#performance-tests).
+monitors, queues, I/O or GC, not only where they use CPU. See
+[Pulsar Performance Testing Framework](#pulsar-performance-testing-framework).
#### Performance recording analysis
-[The performance testing guide](tests/performance/README.md) leads to rendering these recordings into
+[The framework's guide](tests/performance/README.md) leads to rendering these recordings into
flame graphs, opening them in JDK Mission Control or IntelliJ IDEA, analyzing them with an AI agent,
and investigating memory leaks in heap dumps.
@@ -282,17 +283,18 @@ integration tests**, selected with `--tests`, and run the **entire** set with Pe
[`tests/README.md`](tests/README.md) describes running them, selecting TestNG suites and groups, the test
images and profiling a test's cluster.
-### Performance tests
+### Pulsar Performance Testing Framework
-[`tests/performance`](tests/performance/README.md) is for running performance test experiments: it runs
-a Pulsar cluster and its workloads in Docker on one host, as a scenario file describes them, and writes a
-report for every run with throughput, latency, delivery and ordering checks and the host's CPU state.
-Runs can be profiled with async-profiler, JDK Flight Recorder and jonoffcpu's off-CPU recording at the
-same time, and two revisions can be compared. Everything runs from the
-command line and writes its results to files, which makes the experiments automatable, including tuning
-by AI agents. [`tests/performance/README.md`](tests/performance/README.md) is a tutorial for configuring
-a Linux host for consistent results, running a scenario, reading its report, profiling a run and
-comparing revisions.
+The Pulsar Performance Testing Framework, in [`tests/performance`](tests/performance/README.md), is for
+running performance experiments: it runs a Pulsar cluster and its workloads in Docker on one host, as a
+scenario file describes them, and writes a report for every run with throughput, latency, delivery and
+ordering checks and the host's CPU state. Runs can be profiled with async-profiler, JDK Flight Recorder
+and jonoffcpu's off-CPU recording at the same time, and two revisions can be compared, with A/B charts
+of their throughput, backlog and latency. Everything runs from the command line and writes its results
+to files, which makes the experiments automatable, including tuning by AI agents.
+[`tests/performance/README.md`](tests/performance/README.md) is a tutorial for configuring a Linux host
+for consistent results, running a scenario, reading its report, profiling a run and comparing
+revisions.
### Running the full CI pipeline (Personal CI)
@@ -420,5 +422,5 @@ PRs/issues first, then ask via a GitHub issue or dev@pulsar.apache.org.
If you use an AI coding assistant (Claude Code, Copilot, Cursor, Gemini, Codex, Aider, …), see
[`AGENTS.md`](AGENTS.md) for the agent-facing guidance — a routing index into this guide,
[`ARCHITECTURE.md`](ARCHITECTURE.md), [`CODING.md`](CODING.md), and [`SECURITY.md`](SECURITY.md), plus
-the guardrails that apply specifically to AI-made changes. The integration tests, the performance tests
-and the microbenchmarks have agent guides of their own, which `AGENTS.md` links.
+the guardrails that apply specifically to AI-made changes. The integration tests, the Pulsar Performance
+Testing Framework and the microbenchmarks have agent guides of their own, which `AGENTS.md` links.
diff --git a/microbench/AGENTS.md b/microbench/AGENTS.md
index 30be6e961ad8..2e05ca46eec0 100644
--- a/microbench/AGENTS.md
+++ b/microbench/AGENTS.md
@@ -7,4 +7,4 @@ it before running or writing one.
- Run the benchmarks a question needs, selected with a pattern, rather than all of them.
- Treat results from other platforms as provisional until they are confirmed on Linux x86_64, as the README explains.
- A benchmark measures a single class or method. The effect of a change on a running cluster is what
- [the performance tests](../tests/performance/AGENTS.md) measure.
+ [the Pulsar Performance Testing Framework](../tests/performance/AGENTS.md) measures.
diff --git a/microbench/README.md b/microbench/README.md
index 5af9f050b3eb..c3c51b0f1ed1 100644
--- a/microbench/README.md
+++ b/microbench/README.md
@@ -24,8 +24,8 @@
This module contains [JMH](https://github.com/openjdk/jmh) microbenchmarks for Apache Pulsar, for questions about
the performance of a single class or method, such as what a method, a data structure or a codec costs. JMH handles
warm-up, dead-code elimination and measurement, which hand-written timing code gets wrong, so a benchmark is the
-strongest evidence for a small, self-contained optimization. [Performance testing](../tests/performance/README.md)
-covers end-to-end scenarios that run a Pulsar cluster, and profiling and analysis.
+strongest evidence for a small, self-contained optimization.
+[The Pulsar Performance Testing Framework](../tests/performance/README.md) covers end-to-end scenarios that run a Pulsar cluster, and profiling and analysis.
> **Run benchmarks on Linux x86_64 when the numbers matter.** That is Pulsar's most common deployment
> target, and results from elsewhere do not carry over. `System.nanoTime()` is far more expensive on
@@ -147,5 +147,5 @@ Each recording gets a directory beside it named after the file without its exten
(`cpu_classify.html`). A view whose event the recording does not contain is skipped.
The `.jfr` can also be opened in [Eclipse Mission Control](https://adoptium.net/jmc) or IntelliJ
-IDEA, or analyzed by an AI agent. [Performance testing](../tests/performance/README.md) leads to the
-analysis workflow and its tools.
+IDEA, or analyzed by an AI agent.
+[The Pulsar Performance Testing Framework](../tests/performance/README.md) leads to the analysis workflow and its tools.
diff --git a/tests/README.md b/tests/README.md
index 964ddd562a4d..941d25d581ac 100644
--- a/tests/README.md
+++ b/tests/README.md
@@ -21,7 +21,8 @@
# Pulsar tests
-This directory contains the integration tests, the performance tests and the tests of the packaged Pulsar clients.
+This directory contains the integration tests, the Pulsar Performance Testing Framework and the tests of the packaged
+Pulsar clients.
[The microbenchmarks](../microbench/README.md) for single classes and methods are in the `microbench` module.
## Integration tests
@@ -54,10 +55,10 @@ To run the whole suite, use
### Profiling an integration test
> [!NOTE]
-> For performance optimizations, use [the performance tests](performance/README.md) rather than a profiled
-> integration test. They run repeatable scenarios with a report for every run, compare two revisions, and profile
-> where threads wait as well as where they use CPU. Profiling an integration test shows what the cluster of that
-> particular test does.
+> For performance optimizations, use [the Pulsar Performance Testing Framework](performance/README.md) rather than a
+> profiled integration test. It runs repeatable scenarios with a report for every run, compares two revisions, and
+> profiles where threads wait as well as where they use CPU. Profiling an integration test shows what the cluster of
+> that particular test does.
`profilingIntegrationTest` runs an integration test with
[async-profiler](https://github.com/async-profiler/async-profiler) attached to the cluster's components inside their
@@ -88,18 +89,19 @@ be profiled without changing it; name it and the components to profile:
- `-Pdocker.wolfi` builds the images from Wolfi instead of Alpine, which makes the `GLIBC_TUNABLES` that a test sets
take effect.
-Without `--tests`, the task runs `PulsarProfilingTest`, a `pulsar-perf` workload of the legacy TestNG profiling
-runner. [The performance testing guide](performance/README.md) leads to rendering flame graphs from the recordings
+Without `--tests`, the task runs `PulsarProfilingTest`, a `pulsar-perf` workload of the legacy TestNG profiling runner.
+[The Pulsar Performance Testing Framework](performance/README.md) leads to rendering flame graphs from the recordings
and analyzing them, and to the legacy runner.
-## Performance tests
+## Pulsar Performance Testing Framework
-The performance tests are for running performance test experiments. They run a Pulsar cluster and its workloads in
-Docker containers on one host, as a scenario describes them, and write a report for every run: throughput, latency,
-delivery and ordering checks, and the host's CPU state. Runs can be profiled with async-profiler, JDK Flight Recorder
-and jonoffcpu's off-CPU recording at the same time, and two revisions can be compared. The experiments run from the
-command line and write their results to files, so they can be automated, including tuning by AI agents.
-[The performance testing guide](performance/README.md) is a tutorial for getting started.
+The Pulsar Performance Testing Framework is for running performance experiments. It runs a Pulsar cluster and its
+workloads in Docker containers on one host, as a scenario describes them, and writes a report for every run:
+throughput, latency, delivery and ordering checks, and the host's CPU state. Runs can be profiled with async-profiler,
+JDK Flight Recorder and jonoffcpu's off-CPU recording at the same time, and two revisions can be compared, with A/B
+charts of their throughput, backlog and latency. The experiments run from the command line and write their results to
+files, so they can be automated, including tuning by AI agents. [Its guide](performance/README.md) is a tutorial for
+getting started.
## Directories
@@ -114,8 +116,8 @@ command line and write their results to files, so they can be automated, includi
- [`certificate-authority`](certificate-authority/README.md): the test certificate authority and the TLS certificates
that the tests use.
- [`scripts`](scripts): scripts to run before and after the integration tests.
-- [`performance`](performance/README.md): the performance test scenarios, their launcher, the configuration of a
- host for consistent results and the guidance for repeatable profiling runs.
+- [`performance`](performance/README.md): the Pulsar Performance Testing Framework: its scenarios, their launcher, the
+ configuration of a host for consistent results and the guidance for repeatable profiling runs.
- [`compose`](compose/README.md): Docker Compose files that start a Pulsar cluster.
- Tests of the packaged clients: [`pulsar-client-shade-test`](pulsar-client-shade-test),
[`pulsar-client-admin-shade-test`](pulsar-client-admin-shade-test),
diff --git a/tests/docker-images/java-test-image/build.gradle.kts b/tests/docker-images/java-test-image/build.gradle.kts
index f4f106ba2a37..b98baae1b34c 100644
--- a/tests/docker-images/java-test-image/build.gradle.kts
+++ b/tests/docker-images/java-test-image/build.gradle.kts
@@ -118,6 +118,11 @@ fun registerDockerBuild(taskName: String, imageTag: String, installAsyncProfiler
args.add(".")
commandLine(args)
+ // docker writes the image ID file but doesn't create its directory. Gradle creates it for a declared output,
+ // which a build on a released Pulsar image, such as dockerBuildCluster's, doesn't have
+ doFirst {
+ imageIdFile.parentFile.mkdirs()
+ }
inputs.file("Dockerfile")
inputs.files(prepareBuildContext)
diff --git a/tests/integration/AGENTS.md b/tests/integration/AGENTS.md
index 04c41530f465..4daef58ff6ac 100644
--- a/tests/integration/AGENTS.md
+++ b/tests/integration/AGENTS.md
@@ -6,8 +6,8 @@ running or changing them.
- Run single test classes with `--tests`. The full suite is heavy and slow, and runs in CI.
- The tests need Docker. `integrationTest` builds the test image when it is out of date; don't build it separately.
-- For performance optimizations, use the performance tests rather than a profiled integration test, as
- [`tests/performance/AGENTS.md`](../performance/AGENTS.md) guides.
+- For performance optimizations, use the Pulsar Performance Testing Framework rather than a profiled integration
+ test, as [`tests/performance/AGENTS.md`](../performance/AGENTS.md) guides.
## Analysis tools
diff --git a/tests/performance/AGENTS.md b/tests/performance/AGENTS.md
index 7e41dec8048e..557a4f41ac7d 100644
--- a/tests/performance/AGENTS.md
+++ b/tests/performance/AGENTS.md
@@ -1,15 +1,17 @@
-# Agent guide for the performance tests
+# Agent guide for the Pulsar Performance Testing Framework
-Use this guide to turn an experiment into evidence: run a realistic workload, profile it, form a hypothesis,
-change one thing, and compare it with its baseline over repeated runs. The framework runs a Pulsar cluster and
-its workloads in Docker on a developer's machine; a run takes minutes once its images are built. The deliverable
-is a reproducible conclusion with run reports, measurements and an explanation supported by profiles or metrics.
-A faster run is useful only when the workload's delivery guarantees still hold.
+The Pulsar Performance Testing Framework, in `tests/performance`, runs a Pulsar cluster and the client workloads of
+a simulated real-world use case in Docker on a developer's machine, checks the use case's delivery guarantees, and
+writes a report, metrics and, on request, profiles of every run; a run takes minutes once its images are built. Use
+this guide to turn an experiment with it into evidence: run a realistic workload, profile it, form a hypothesis,
+change one thing, and compare it with its baseline over repeated runs. The deliverable is a reproducible conclusion
+with run reports, measurements and an explanation supported by profiles or metrics. A faster run is useful only when
+the workload's delivery guarantees still hold.
-This supplements the repository's [`AGENTS.md`](../../AGENTS.md). [`README.md`](README.md) is the tutorial;
-read it before running or changing the tests. This guide is the agent's working reference, and the linked pages
-describe the full interfaces. There is no automatic baseline comparison or performance regression verdict:
-the agent must make and explain the comparison.
+This supplements the repository's [`AGENTS.md`](../../AGENTS.md). [`README.md`](README.md) is the tutorial; read it
+before running or changing the framework. This guide is the agent's working reference, and the linked pages describe
+the full interfaces. The framework charts a comparison but gives no performance verdict: the agent must make and
+explain the comparison.
The testing strategy is to simulate real-world use cases of Pulsar: a domain models a use case, currently IoT
telemetry, and each scenario sets its scale so that it maps to a kind of real-world deployment. This keeps the tests
@@ -18,6 +20,109 @@ real deployments use together and checks their delivery guarantees, so its impro
real-world use. Express experiments and new scenarios in the domain's terms, as
[How the tests work](README.md#how-the-tests-work) describes.
+## Before running
+
+A run takes minutes and uses the whole host. Don't start runs, or run them alongside other runs or builds, unless the
+user asked for them. Before a series of runs, check these:
+
+| Check | How | When it fails |
+|---|---|---|
+| Memory available to Docker. Recommended headroom: 20 GB for Docker and 32 GB on the host | Scenario memory configurations: low about 3 GB; medium (default) about 11 GB; high about 14 GB; see [Memory configurations](scenarios/README.md#memory-configurations) | Tell the user if the scenario will not fit; the low-memory scenarios are for smaller runs, not profiling |
+| Docker is available and has sufficient disk space; on Linux also fixed CPU frequency, stopped power-management daemons and AC power | [`configure-perf-test-environment.sh`](environment/scripts/configure-perf-test-environment.sh) `validate` checks Docker and disk on Linux and macOS, and host tuning only on Linux; failed checks print to stderr | Find the failed checks from its exit code in the table of [Exit codes](environment/README.md#exit-codes), and ask the user before doing what it [...]
+| Profiled scenario | Profile a scenario with the medium- or the high-memory configuration | Don't profile a scenario that uses the low-memory configuration |
+
+Follow the repository's [build prerequisites](../../CONTRIBUTING.md#building) for Java and Docker.
+`docker compose version` checks the Compose plugin needed by metrics. Collection is enabled by default: a run
+reuses a running stack, or starts and stops one for itself. Missing metrics or failed panel rendering may only
+produce a warning, so check `metrics.json` and the report before relying on that evidence.
+
+Profiling also changes the environment: `profile` depends on `:tests:integration:tuneKernelPerfEvents`, which
+runs a privileged container to change perf-event, BPF and transparent-huge-page settings in the Docker engine's
+Linux kernel (the VM's kernel on macOS). Profiled containers run privileged with their JVM as root. Account for
+this when preparing the host; these are not just image-build steps. When the kernel is already configured,
+`-Pinttest.asyncprofiler.skipPerfEventTuning` skips the tuning task; the environment script's `start` sets that
+property until `stop`. See [Profiling requirements](docs/profiling.md#requirements).
+
+Variance sets the smallest change that a comparison can detect: when the spread between runs of the same revision is
+larger than a change's effect, the effect can't be told apart from noise. When you'll be running experiments
+repeatedly on Linux, you can suggest that the user sets up the sudoers rule that
+[Running start and stop without a password](environment/README.md#running-start-and-stop-without-a-password)
+describes. It lets you run `sudo /usr/local/sbin/configure-perf-test-environment.sh start` before the runs and `stop`
+after them yourself; `install` still needs the user.
+On a host with Linux temperature sensors, the optional cool-down gate can reduce thermal drift when fixed-frequency
+tuning is unavailable; it does not replace checking throttling or run-to-run spread. It proceeds after
+`--cooldown-timeout` (600 seconds by default), so inspect the report's Host section for whether the target was reached.
+
+### Platforms
+
+The cluster and the workloads run in Linux containers, so the tests run on any host with a Docker engine. The
+README's [Before you start](README.md#before-you-start) lists the hosts that the tooling was tested on.
+
+Linux x86_64 is Pulsar's main target platform, and on dedicated hardware a run has no noisy neighbours and less
+thermal and power throttling and CPU frequency variance. On macOS and Windows, Docker runs in a virtual machine that
+shares the host's CPUs, memory, disk and network with the host operating system, which schedules them, so the results
+aren't representative of a Linux deployment and vary more between runs. Off-CPU profiling needs a kernel with BTF,
+see [Profiling](docs/profiling.md#requirements); without it, profile with async-profiler and JDK Flight Recorder
+only. Non-Linux hosts also lack the launcher's `host-stats.csv` evidence of temperature and throttling; a passing
+Docker/disk validation does not fill that gap. Use non-Linux comparisons as exploratory results for that host:
+name the host and Docker engine, repeat runs to establish their spread, and do not present the result as
+representative of a Linux deployment. Confirm a deployment-performance claim on Linux x86_64. Never compare
+runs made on different hosts.
+
+## Quick reference
+
+Run commands from the repository root. Gradle properties (`-P...`) configure the task; launcher options go inside
+`--args='...'`. The launcher tasks build their images and workload applications when needed, then run the experiment.
+
+| Purpose | Command or input | Output / effect |
+|---|---|---|
+| Run an unprofiled measurement | `./gradlew :tests:performance:launcher:run --args='--scenario tests/performance/scenarios/iot-telemetry.yaml'` | Prints the run directory, resolved settings, progress and report path; writes the [run artifacts](#the-run) |
+| Profile selected components | `./gradlew :tests:performance:launcher:profile --args='--scenario tests/performance/scenarios/iot-telemetry.yaml --extends configs/profile-broker'` | Adds JFR recordings, off-CPU captures, digests and flame graphs; use `profile`, since `run` rejects profiling options |
+| Select more profiled components | Repeat `--extends configs/profile-gateways` and/or `--extends configs/profile-applications` inside `--args` of `profile` | Profiles producers and/or consumers; each workload container's recording covers all its gateways or applications |
+| Profile Netty's buffer allocations | Use `--extends configs/profile-<component>-netty-allocations` in place of `configs/profile-<component>` | Records Netty's allocator events (`jfrConfigurations: [profile, netty-allocations.jfc]`) and summarizes them in the profile report (`nettyAllocationsReport: true`). Heavy overhead: an event for every buffer, about 2.5 per message of each kind at max rate and a 1 GB broker recording, so use a short measurement, don't compare its throughput or lat [...]
+| Change a setting for one run | Add `--set workloads.iotTelemetry.rate=5000` to `--args` | Applies after inheritance and environment overrides; inspect `resolved-config.yaml` to verify it |
+| Group an experiment's runs | Add `--name <experiment>` to `--args` | Uses that name below the checkout's branch directory, which can be shared by both revisions; does not combine or compare reports |
+| Keep revisions' images apart | Add `-Pdocker.tag=baseline` or `-Pdocker.tag=candidate` | Builds and uses a separate image tag in each checkout |
+| Share a reports root | Add `-Pperformance.reportsDir=<absolute-directory>` to run and report-server commands | Keeps evidence outside worktrees; alternatively set `performance.reportsDir=/absolute/path` once in `~/.gradle/gradle.properties` |
+| Keep logs for diagnosis | Add `-Pperformance.keepLauncherLog` | Retains `launcher.log` even on success; failures retain it by default |
+| Start repetitions at a similar temperature | Add `-Pperformance.cooldownTemperature=<degrees-C>` | With Linux host sensors, waits before cluster startup and after warmup; keep the threshold identical across revisions and inspect timed-out waits in the report |
+| Disable metrics | Add `-Pperformance.metrics=false` | Omits metrics collection; keep this choice identical across compared runs |
+| Chart two runs against each other | `./gradlew :tests:performance:report-tool:compareRuns --args='--baseline <A run> --comparison <B run> --output <directory>'` | Writes throughput, backlog and latency charts with the same axes for both runs, combined and in separate panels, as [Comparison charts](docs/comparing-revisions.md#comparison-charts) describes |
+| Browse reports | `./gradlew :tests:performance:report-tool:serveReports` | Serves the configured reports root at <http://127.0.0.1:8000/> by default; stop the foreground server when finished |
+| Query stored metrics after a run | `./gradlew :tests:performance:metrics:up` | Starts VictoriaMetrics and Grafana in the background, prints URLs; default ports are 8428 and 3000 |
+| Stop that metrics stack | `./gradlew :tests:performance:metrics:down` | Stops containers while retaining metrics and dashboards in Docker volumes |
+| Validate the host | `tests/performance/environment/scripts/configure-perf-test-environment.sh validate` | Checks Docker and disk space; on Linux also host tuning and AC power; errors go to stderr with a documented exit code |
+
+[Running scenarios](docs/running-scenarios.md) lists every launcher option and Gradle property. A profiled
+component's `asyncProfilerOptions` is a comma-separated list of
+[async-profiler's options](https://github.com/async-profiler/async-profiler/blob/master/docs/ProfilerOptions.md), as
+the page's "Launch as agent" column names them, and
+[Configuring profiling](docs/profiling.md#configuring-profiling) describes its other settings.
+Relative `--extends` paths resolve against the scenario file's directory first, then the working directory;
+`.yaml` can be omitted. Use an absolute path for a configuration outside those locations.
+`./gradlew :tests:performance:launcher:run --args='--help'` prints the launcher's options, but its Gradle image-build
+dependencies still run; it is not a side-effect-free preflight command.
+
+For an A/B run, use this in the baseline checkout and repeat it in the candidate checkout with
+`-Pdocker.tag=candidate`. Alternate them for the repetitions; do not run the commands concurrently:
+
+```bash
+./gradlew :tests:performance:launcher:run -Pdocker.tag=baseline \
+ -Pperformance.reportsDir="$HOME/pulsar-performance-reports" \
+ --args='--scenario tests/performance/scenarios/iot-telemetry.yaml --name my-change-ab'
+```
+
+Use the printed `Run directory:` as the exact run identity. By default the layout is
+`<reports root>/<yyyy-MM-dd>/<branch>/<name>/<MM-dd-HH-mm-ss>/`, under `build/performance` if no root is configured.
+`Run report:` names `index.html`; read the sibling `README.md` for text analysis. A report URL does not start the
+HTTP server. Progress lines appear every 10 seconds and are copied to `console.log.txt`; they include warmup and
+interval statistics, so use the report and summary files for measurement results. Avoid selecting a run merely
+because it is the newest: associate each command, revision and resolved configuration with its printed path.
+Check the commit and dirty state in `run-info.json`: a detached baseline can share the candidate's branch directory.
+The report's delivered throughput divides measured messages by the time until the slowest application received
+the last one; console receive rates and broker dispatch metrics sum deliveries across subscriptions. Do not
+compare those as if they were the same measure.
+
## Experiment loop
1. **Define the question.** Record the baseline and candidate revisions, the scenario and overrides, the expected
@@ -50,6 +155,61 @@ real-world use. Express experiments and new scenarios in the domain's terms, as
source control. Restore host settings and stop services that this experiment started, within the user's
authorization; leave a pre-existing metrics stack running.
+## Using a run's results
+
+- Find a run's results from the launcher's output, which prints the run directory and the run report.
+- Use a run's measurement results only when its report shows a valid run, as [Read the report](README.md#2-read-the-report)
+ describes: every application received every message, without ordering violations or invalid messages. Duplicates
+ are counted and allowed by at-least-once delivery; the report flags them, but they do not alone fail the run.
+ Report and investigate a change in duplicates rather than assuming the warning means a delivery failure.
+- Don't claim a performance change from a single run: compare revisions as
+ [Comparing revisions](docs/comparing-revisions.md) describes.
+- A run that fails writes no report. Find the cause as [When a run fails](docs/run-reports.md#when-a-run-fails)
+ describes, and tell the user rather than using the run.
+- Don't use the measurement results of a run that wrote heap dumps: the dumps stop the JVM.
+- Don't use a profiled run's measurement results for latency and throughput improvement claims; use comparisons
+ between matching profiled runs to investigate the cause instead.
+- Exclude runs with timeouts, out-of-memory errors, broker restarts or incomparable host throttling, even if clients
+ recovered. Keep diagnostic evidence and record why the run was excluded. Missing host samples are not evidence
+ that the host did not throttle.
+- Don't add run output to the git repository.
+
+## Comparison charts
+
+Chart a comparison with `compareRuns` once both revisions have valid runs. Its charts show a baseline run (A) and a
+comparison run (B) with the same axes, so that their values can be compared by position; the run reports' own
+charts each have their own scale. Use them in experiment notes, pull requests and documentation.
+
+1. **Pick the runs.** Use each revision's median run by the measure that the comparison is about, such as the
+ steady consumption rate or a latency percentile, from valid unprofiled runs of the same scenario and options. Say
+ which runs they are.
+2. **Render the charts.** Give each run a short label that names what differs, such as a release version, a branch
+ or a setting; the labels appear in the charts' legends, panel badges and file names:
+
+ ```bash
+ ./gradlew :tests:performance:report-tool:compareRuns --args="--baseline <A run directory> \
+ --comparison <B run directory> --baseline-label 4.0.13 --comparison-label 5.0.0 --output <directory>"
+ ```
+
+ A label defaults to its run's name. `--no-labels-in-file-names` leaves the labels out of the file names, for
+ example when the directory already names the comparison.
+3. **Choose the charts.** For the throughput (published and consumed), the backlog, and the latency by percentile
+ on a linear and on a logarithmic axis (`latency-percentiles-log`), it writes two forms:
+ - `<chart>-<A label>-vs-<B label>.svg`: both runs in one diagram, A in blue and B in orange;
+ - `<chart>-<A label>-vs-<B label>-separate.svg`: A above B in panels with the same axes, tinted in the runs'
+ colors, which reads better when the lines overlap and for readers new to the charts.
+
+ Prefer the logarithmic latency chart when the runs' latencies differ by orders of magnitude, which flattens one of
+ them on a linear axis. Markers in each run's color show when its gateways finished publishing and when its
+ consumers received their last message.
+4. **Look at the charts before using them.** They are SVG: convert one to view it, for example with
+ `rsvg-convert -w 1400 <chart>.svg -o <chart>.png`, and check the labels, the markers and that the axes cover both
+ runs.
+5. **Report the numbers too.** The charts show one run of each revision; the table of medians and spreads that
+ [Experiment handoff](#experiment-handoff) describes is the result.
+
+[Comparison charts](docs/comparing-revisions.md#comparison-charts) describes the charts and the task's options.
+
## Eliminating bottlenecks
Use this mode when the task is to raise a scenario's capacity rather than to validate one change. A system has one
@@ -106,130 +266,27 @@ Lessons from earlier iterations:
- The launcher doesn't profile the bookies. Their metrics (`bookie_throttled_write`, `bookie_flush`, journal queue
and flush latencies, `bookkeeper_server_ADD_ENTRY*`) and their threads' CPU, sampled from `/proc/<pid>/task/*/stat`
inside the container with `docker exec`, show whether a bookie thread or its storage is the limit.
+- **When the disk is the limit.** All the bookies share the host's disk, and on a developer machine it is often a
+ laptop SSD, sometimes behind full-disk encryption, that limits large unbatched entries long before Pulsar does. The
+ signs: the disk's write MB/s in `host-io.csv` stays flat across runs and variants while the host's CPU has headroom;
+ `bookie_throttled_write` counts writes; the broker's managed-ledger add latency is high while the bookies' own add
+ latency is low and no broker or bookie thread is busy. Two options take the disk out of the way, so that the run
+ measures Pulsar instead:
+ - [`--extends configs/bookie-journal-tmpfs`](scenarios/README.md#the-bookies-journal-on-a-tmpfs) puts the bookies'
+ journals on a tmpfs. Every entry is written to the disk twice, once to the journal and once to the entry log, so
+ this halves the disk's writes. The test cluster doesn't sync the journal anyway, so it changes no durability
+ guarantee of the test; it costs about 512 MB of memory per busy bookie.
+ - [`configure-perf-test-environment.sh start --disable-write-barriers`](environment/README.md#disabling-write-barriers)
+ makes the ledger storage's syncs at each write-cache flush stop waiting for the disk's volatile cache. It risks
+ the host's file system on a power loss, so use it only on a host where that is acceptable, and only when the
+ tmpfs alone isn't enough.
+
+ Neither changes a limit inside a bookie: with one topic on single-copy ledgers, one bookie's single write-cache
+ flush thread (`db-storage`) still caps large entries (about 500 MB/s on an i9-9980HK). Compare runs only with the
+ same options, and mention them with the results.
Before proposing a change for review, validate it as the [Experiment loop](#experiment-loop) describes.
-## Quick reference
-
-Run commands from the repository root. Gradle properties (`-P...`) configure the task; launcher options go inside
-`--args='...'`. The launcher tasks build their images and workload applications when needed, then run the experiment.
-
-| Purpose | Command or input | Output / effect |
-|---|---|---|
-| Run an unprofiled measurement | `./gradlew :tests:performance:launcher:run --args='--scenario tests/performance/scenarios/iot-telemetry.yaml'` | Prints the run directory, resolved settings, progress and report path; writes the [run artifacts](#the-run) |
-| Profile selected components | `./gradlew :tests:performance:launcher:profile --args='--scenario tests/performance/scenarios/iot-telemetry.yaml --extends configs/profile-broker'` | Adds JFR recordings, off-CPU captures, digests and flame graphs; use `profile`, since `run` rejects profiling options |
-| Select more profiled components | Repeat `--extends configs/profile-gateways` and/or `--extends configs/profile-applications` inside `--args` of `profile` | Profiles producers and/or consumers; each workload container's recording covers all its gateways or applications |
-| Profile Netty's buffer allocations | Use `--extends configs/profile-<component>-netty-allocations` in place of `configs/profile-<component>` | Records Netty's allocator events (`jfrConfigurations: [profile, netty-allocations.jfc]`) and summarizes them in the profile report (`nettyAllocationsReport: true`). Heavy overhead: an event for every buffer, about 2.5 per message of each kind at max rate and a 1 GB broker recording, so use a short measurement, don't compare its throughput or lat [...]
-| Change a setting for one run | Add `--set workloads.iotTelemetry.rate=5000` to `--args` | Applies after inheritance and environment overrides; inspect `resolved-config.yaml` to verify it |
-| Group an experiment's runs | Add `--name <experiment>` to `--args` | Uses that name below the checkout's branch directory, which can be shared by both revisions; does not combine or compare reports |
-| Keep revisions' images apart | Add `-Pdocker.tag=baseline` or `-Pdocker.tag=candidate` | Builds and uses a separate image tag in each checkout |
-| Share a reports root | Add `-Pperformance.reportsDir=<absolute-directory>` to run and report-server commands | Keeps evidence outside worktrees; alternatively set `performance.reportsDir=/absolute/path` once in `~/.gradle/gradle.properties` |
-| Keep logs for diagnosis | Add `-Pperformance.keepLauncherLog` | Retains `launcher.log` even on success; failures retain it by default |
-| Start repetitions at a similar temperature | Add `-Pperformance.cooldownTemperature=<degrees-C>` | With Linux host sensors, waits before cluster startup and after warmup; keep the threshold identical across revisions and inspect timed-out waits in the report |
-| Disable metrics | Add `-Pperformance.metrics=false` | Omits metrics collection; keep this choice identical across compared runs |
-| Browse reports | `./gradlew :tests:performance:report-tool:serveReports` | Serves the configured reports root at <http://127.0.0.1:8000/> by default; stop the foreground server when finished |
-| Query stored metrics after a run | `./gradlew :tests:performance:metrics:up` | Starts VictoriaMetrics and Grafana in the background, prints URLs; default ports are 8428 and 3000 |
-| Stop that metrics stack | `./gradlew :tests:performance:metrics:down` | Stops containers while retaining metrics and dashboards in Docker volumes |
-| Validate the host | `tests/performance/environment/scripts/configure-perf-test-environment.sh validate` | Checks Docker and disk space; on Linux also host tuning and AC power; errors go to stderr with a documented exit code |
-
-[Running scenarios](docs/running-scenarios.md) lists every launcher option and Gradle property. A profiled
-component's `asyncProfilerOptions` is a comma-separated list of
-[async-profiler's options](https://github.com/async-profiler/async-profiler/blob/master/docs/ProfilerOptions.md), as
-the page's "Launch as agent" column names them, and
-[Configuring profiling](docs/profiling.md#configuring-profiling) describes its other settings.
-Relative `--extends` paths resolve against the scenario file's directory first, then the working directory;
-`.yaml` can be omitted. Use an absolute path for a configuration outside those locations.
-`./gradlew :tests:performance:launcher:run --args='--help'` prints the launcher's options, but its Gradle image-build
-dependencies still run; it is not a side-effect-free preflight command.
-
-For an A/B run, use this in the baseline checkout and repeat it in the candidate checkout with
-`-Pdocker.tag=candidate`. Alternate them for the repetitions; do not run the commands concurrently:
-
-```bash
-./gradlew :tests:performance:launcher:run -Pdocker.tag=baseline \
- -Pperformance.reportsDir="$HOME/pulsar-performance-reports" \
- --args='--scenario tests/performance/scenarios/iot-telemetry.yaml --name my-change-ab'
-```
-
-Use the printed `Run directory:` as the exact run identity. By default the layout is
-`<reports root>/<yyyy-MM-dd>/<branch>/<name>/<MM-dd-HH-mm-ss>/`, under `build/performance` if no root is configured.
-`Run report:` names `index.html`; read the sibling `README.md` for text analysis. A report URL does not start the
-HTTP server. Progress lines appear every 10 seconds and are copied to `console.log.txt`; they include warmup and
-interval statistics, so use the report and summary files for measurement results. Avoid selecting a run merely
-because it is the newest: associate each command, revision and resolved configuration with its printed path.
-Check the commit and dirty state in `run-info.json`: a detached baseline can share the candidate's branch directory.
-The report's delivered throughput divides measured messages by the time until the slowest application received
-the last one; console receive rates and broker dispatch metrics sum deliveries across subscriptions. Do not
-compare those as if they were the same measure.
-
-## Before running
-
-A run takes minutes and uses the whole host. Don't start runs, or run them alongside other runs or builds, unless the
-user asked for them. Before a series of runs, check these:
-
-| Check | How | When it fails |
-|---|---|---|
-| Memory available to Docker. Recommended headroom: 20 GB for Docker and 32 GB on the host | Scenario memory configurations: low about 3 GB; medium (default) about 11 GB; high about 14 GB; see [Memory configurations](scenarios/README.md#memory-configurations) | Tell the user if the scenario will not fit; the low-memory scenarios are for smaller runs, not profiling |
-| Docker is available and has sufficient disk space; on Linux also fixed CPU frequency, stopped power-management daemons and AC power | [`configure-perf-test-environment.sh`](environment/scripts/configure-perf-test-environment.sh) `validate` checks Docker and disk on Linux and macOS, and host tuning only on Linux; failed checks print to stderr | Find the failed checks from its exit code in the table of [Exit codes](environment/README.md#exit-codes), and ask the user before doing what it [...]
-| Profiled scenario | Profile a scenario with the medium- or the high-memory configuration | Don't profile a scenario that uses the low-memory configuration |
-
-Follow the repository's [build prerequisites](../../CONTRIBUTING.md#building) for Java and Docker.
-`docker compose version` checks the Compose plugin needed by metrics. Collection is enabled by default: a run
-reuses a running stack, or starts and stops one for itself. Missing metrics or failed panel rendering may only
-produce a warning, so check `metrics.json` and the report before relying on that evidence.
-
-Profiling also changes the environment: `profile` depends on `:tests:integration:tuneKernelPerfEvents`, which
-runs a privileged container to change perf-event, BPF and transparent-huge-page settings in the Docker engine's
-Linux kernel (the VM's kernel on macOS). Profiled containers run privileged with their JVM as root. Account for
-this when preparing the host; these are not just image-build steps. When the kernel is already configured,
-`-Pinttest.asyncprofiler.skipPerfEventTuning` skips the tuning task; the environment script's `start` sets that
-property until `stop`. See [Profiling requirements](docs/profiling.md#requirements).
-
-Variance sets the smallest change that a comparison can detect: when the spread between runs of the same revision is
-larger than a change's effect, the effect can't be told apart from noise. When you'll be running experiments
-repeatedly on Linux, you can suggest that the user sets up the sudoers rule that
-[Running start and stop without a password](environment/README.md#running-start-and-stop-without-a-password)
-describes. It lets you run `sudo /usr/local/sbin/configure-perf-test-environment.sh start` before the runs and `stop`
-after them yourself; `install` still needs the user.
-On a host with Linux temperature sensors, the optional cool-down gate can reduce thermal drift when fixed-frequency
-tuning is unavailable; it does not replace checking throttling or run-to-run spread. It proceeds after
-`--cooldown-timeout` (600 seconds by default), so inspect the report's Host section for whether the target was reached.
-
-### Platforms
-
-The cluster and the workloads run in Linux containers, so the tests run on any host with a Docker engine. The
-README's [Before you start](README.md#before-you-start) lists the hosts that the tooling was tested on.
-
-Linux x86_64 is Pulsar's main target platform, and on dedicated hardware a run has no noisy neighbours and less
-thermal and power throttling and CPU frequency variance. On macOS and Windows, Docker runs in a virtual machine that
-shares the host's CPUs, memory, disk and network with the host operating system, which schedules them, so the results
-aren't representative of a Linux deployment and vary more between runs. Off-CPU profiling needs a kernel with BTF,
-see [Profiling](docs/profiling.md#requirements); without it, profile with async-profiler and JDK Flight Recorder
-only. Non-Linux hosts also lack the launcher's `host-stats.csv` evidence of temperature and throttling; a passing
-Docker/disk validation does not fill that gap. Use non-Linux comparisons as exploratory results for that host:
-name the host and Docker engine, repeat runs to establish their spread, and do not present the result as
-representative of a Linux deployment. Confirm a deployment-performance claim on Linux x86_64. Never compare
-runs made on different hosts.
-
-## Using a run's results
-
-- Find a run's results from the launcher's output, which prints the run directory and the run report.
-- Use a run's measurement results only when its report shows a valid run, as [Read the report](README.md#2-read-the-report)
- describes: every application received every message, without ordering violations or invalid messages. Duplicates
- are counted and allowed by at-least-once delivery; the report flags them, but they do not alone fail the run.
- Report and investigate a change in duplicates rather than assuming the warning means a delivery failure.
-- Don't claim a performance change from a single run: compare revisions as
- [Comparing revisions](docs/comparing-revisions.md) describes.
-- A run that fails writes no report. Find the cause as [When a run fails](docs/run-reports.md#when-a-run-fails)
- describes, and tell the user rather than using the run.
-- Don't use the measurement results of a run that wrote heap dumps: the dumps stop the JVM.
-- Don't use a profiled run's measurement results for latency and throughput improvement claims; use comparisons
- between matching profiled runs to investigate the cause instead.
-- Exclude runs with timeouts, out-of-memory errors, broker restarts or incomparable host throttling, even if clients
- recovered. Keep diagnostic evidence and record why the run was excluded. Missing host samples are not evidence
- that the host did not throttle.
-- Don't add run output to the git repository.
-
## Inputs for analysis in a run directory
Prefer text artifacts — Markdown, JSON, CSV and collapsed stacks — to HTML pages, which need a browser to render.
@@ -426,16 +483,16 @@ run directories and any per-recording analyses. Give the user:
which direction is better. State the number of valid repetitions. Report a latency percentile's median across
runs as such; do not present it as the percentile of pooled messages. A difference within run-to-run spread
is inconclusive, and three runs alone do not establish statistical significance.
-- Links to the reports, relevant charts, profile callsites and metric queries that support the explanation,
- alongside limitations, missing evidence and the next experiment if the question remains open.
+- The [comparison charts](#comparison-charts) of the median runs, and links to the reports, profile callsites and
+ metric queries that support the explanation, alongside limitations, missing evidence and the next experiment if
+ the question remains open.
## Pull requests for performance improvements
When the user asks you to create a pull request, or update one, for a performance improvement that was tested with
-these tests, suggest adding the charts of the comparison to the pull request's description as SVG images. Each run
-directory has its charts as SVG, such as `throughput.svg` and `latency-percentiles.svg`, see
-[Files of a run](docs/run-reports.md#files-of-a-run). Use the run of each revision whose numbers are closest to that
-revision's medians, and suggest the charts that show the improvement. Add the charts only when the user agrees.
+the framework, suggest adding charts of the comparison to the pull request's description as SVG images: render them
+as [Comparison charts](#comparison-charts) describes, and suggest the ones that show the improvement. Add the charts
+only when the user agrees.
These instructions are for the `apache/pulsar` repository, when a human has prepared the change and asks you to
create the pull request. Never open or update a pull request in `apache/pulsar` without the human's explicit
@@ -447,14 +504,15 @@ personal fork or another fork.
- Attach the charts with the `--attach` option of `gh pr create` and `gh pr edit`, which recent versions of the
GitHub CLI have. Check its details with `gh pr create --help` or `gh pr edit --help`, and if the installed `gh`
lacks the option, tell the user.
-- The charts of every run have the same file names, so copy them to names that say the revision, such as
- `baseline-throughput.svg` and `candidate-throughput.svg`, before attaching them.
-- Give each image an alt text that says what the chart shows and for which revision:
- `--attach './candidate-throughput.svg#Throughput with the change'`.
+- The comparison charts' file names already say both revisions. A run report's own charts, such as `throughput.svg`
+ of [Files of a run](docs/run-reports.md#files-of-a-run), have the same names in every run: copy them to names that
+ say the revision, such as `baseline-throughput.svg`, before attaching them.
+- Give each image an alt text that says what the chart shows and for which revisions:
+ `--attach './throughput-master-vs-change-separate.svg#Throughput of master and the change'`.
- `gh` appends an attachment to the end of the description, unless the description references the file with a
- Markdown image, such as ``, in which case it points that
- reference at the uploaded file. Put the references where the charts belong, such as in a table with a column for
- the baseline and one for the change.
+ Markdown image, such as ``, in
+ which case it points that reference at the uploaded file. Put the references where the charts belong, such as
+ under the table of measurements.
- Then turn each Markdown image into an HTML image with a width, wrapped in a link to the uploaded file, by editing the
description, for example with `gh pr view --json body` and `gh pr edit --body-file`:
`<a href="<uploaded URL>"><img src="<uploaded URL>" alt="Throughput with the change" width="480"></a>`. Markdown
@@ -463,7 +521,7 @@ personal fork or another fork.
- The charts don't replace the numbers: state the medians of the measures and their changes in the text, as
[Compare](docs/comparing-revisions.md#compare) describes.
-## Changing the performance tests
+## Changing the framework
- New scenarios, workload applications and profiling support belong to the standalone launcher, not to the
deprecated TestNG runner under `tests/integration`.
diff --git a/tests/performance/README.md b/tests/performance/README.md
index c96f968e7561..0cd94e845fee 100644
--- a/tests/performance/README.md
+++ b/tests/performance/README.md
@@ -19,12 +19,12 @@
-->
-# Performance testing
+# Pulsar Performance Testing Framework
-The performance tests are for running performance test experiments: measuring a change, comparing two revisions
-and finding what to optimize. They run a Pulsar cluster and its client workloads in Docker containers on one host, as
-a [scenario](scenarios/README.md) describes them, and write a report for every run: throughput, latency, delivery and
-ordering checks, and the host's CPU temperature. A run can also be profiled with
+The Pulsar Performance Testing Framework is for running performance experiments: measuring a change, comparing two
+revisions and finding what to optimize. It runs a Pulsar cluster and its client workloads in Docker containers on one
+host, as a [scenario](scenarios/README.md) describes them, and writes a report for every run: throughput, latency,
+delivery and ordering checks, and the host's CPU temperature. A run can also be profiled with
[async-profiler](https://github.com/async-profiler/async-profiler),
[JDK Flight Recorder](https://docs.oracle.com/en/java/javase/25/troubleshoot/diagnostic-tools.html#GUID-D38849B6-61C7-4ED6-A395-EA4BC32A9FD6)
and [jonoffcpu](https://github.com/jonoffcpu/jonoffcpu) at the same time, which gives CPU, allocation and off-CPU
@@ -32,9 +32,9 @@ flame graphs of the broker and the clients: both where threads use CPU and where
command line and writes its results to files, so that experiments can be automated, including tuning by AI agents,
which [`AGENTS.md`](AGENTS.md) guides.
-The performance tests aren't currently used as automated regression tests: no CI job runs the scenarios, and no run
-is checked against a baseline automatically. A person or an agent compares revisions, as
-[Compare two revisions](#5-compare-two-revisions) describes. A future improvement is to evolve the tests into an
+The framework isn't currently used for automated regression testing: no CI job runs the scenarios, and no run is
+checked against a baseline automatically. A person or an agent compares revisions, as
+[Compare two revisions](#5-compare-two-revisions) describes. A future improvement is to evolve the framework into an
automated regression test suite, which would prevent performance regressions by checking changes against a baseline,
and set the new baseline when a change improves performance.
@@ -130,7 +130,8 @@ scenarios in detail.
| [Podman Desktop](https://podman-desktop.io/) | | Untested |
- **Memory**: 32 GB of RAM on the host is recommended, although testing may be possible with less. A scenario's memory
configuration sets the heap and direct memory of the cluster's and the workloads' JVMs, and so the memory that the
- host has to have available to Docker, which on macOS and Windows is the memory of Docker's virtual machine. The IoT telemetry scenarios use the medium-memory configuration by default; see
+ host has to have available to Docker, which on macOS and Windows is the memory of Docker's virtual machine. The IoT
+ telemetry scenarios use the medium-memory configuration by default; see
[Memory configurations](scenarios/README.md#memory-configurations):
| Configuration | Recommended memory available to Docker | Used by |
@@ -141,6 +142,10 @@ scenarios in detail.
- **Disk space**: keep the disk that holds Docker's data less than 90 % full. BookKeeper bookies switch to read-only
mode when it is 95 % full. [`docker-cleanup.sh`](environment/scripts/docker-cleanup.sh) frees the space that test
runs and image builds use up.
+- **Disk speed**: all the bookies write to the host's disk, which can limit runs with large messages before Pulsar
+ does. The [bookies' journal on a tmpfs](scenarios/README.md#the-bookies-journal-on-a-tmpfs) halves its writes, and
+ [disabling write barriers](environment/README.md#disabling-write-barriers) makes its syncs cheaper on a disk without
+ power loss protection.
- **Recommended: A Linux host configured for consistent results**: turbo frequencies depend on the CPU's temperature,
and power management changes CPU settings during a run, so results vary between runs of the same code, and a change
smaller than that variance can't be detected. [The performance testing environment setup](environment/README.md) fixes
@@ -329,8 +334,8 @@ The `profile` task runs a scenario with three recorders running at the same time
records the JVM's own events into the same recording, and
[jonoffcpu](https://github.com/jonoffcpu/jonoffcpu) records from the kernel the time each thread spent blocked. It
needs a Docker engine whose kernel has BTF, such as a Linux host's or [OrbStack](https://orbstack.dev/)'s on macOS, see
-[Requirements](docs/profiling.md#requirements). The profiling scenario publishes 30,000 messages per second to one topic from 500
-producers, and needs about 14 GB of memory available to Docker:
+[Requirements](docs/profiling.md#requirements). The profiling scenario publishes 30,000 messages per second to one
+topic from 500 producers, and needs about 14 GB of memory available to Docker:
```bash
./gradlew :tests:performance:launcher:profile \
@@ -381,7 +386,8 @@ inferno-flamegraph < /path/to/recording-offcpu/offcpu-no-idle-app-root.collapsed
```
For SQL analysis, DuckDB's [quack_flamegraph](https://github.com/kevintruong/quack-flamegraph) community extension
-reads collapsed stacks as tables. See [Analysing collapsed stacktrace files with DuckDB and quack_flamegraph](docs/analyzing-profiles.md#analysing-collapsed-stacktrace-files-with-quack_flamegraph)
+reads collapsed stacks as tables. See
+[Analyzing collapsed stacktrace files with quack_flamegraph](docs/analyzing-profiles.md#analyzing-collapsed-stacktrace-files-with-quack_flamegraph)
for setup and examples that rank stacks, methods and call edges, and attribute samples to application frames.
The JFR recordings also open in JDK Mission Control, whose OpenJDK distribution is
@@ -425,6 +431,17 @@ such as the latest release or a particular one. The workloads, and so the Pulsar
[Comparing with a released Pulsar](docs/comparing-revisions.md#comparing-with-a-released-pulsar) describes the
details.
+When both revisions have valid runs, chart the median run of each against the other, with the same axes for both
+and each revision's label in the charts and their file names:
+
+```bash
+./gradlew :tests:performance:report-tool:compareRuns --args="--baseline <baseline run directory> \
+ --comparison <candidate run directory> --baseline-label baseline --comparison-label candidate --output <directory>"
+```
+
+[Comparison charts](docs/comparing-revisions.md#comparison-charts) describes the throughput, backlog and latency
+charts that it writes, in one diagram and in separate panels.
+
## Reference
- [Running scenarios](docs/running-scenarios.md): the Gradle tasks, the launcher's options and Gradle properties,
@@ -433,20 +450,21 @@ details.
HTTP.
- [Scenarios](scenarios/README.md) and [the scenario format](scenarios/docs/scenario-format.md): the maintained
scenarios, inheritance, environment overrides and the warmup.
-- [Profiling](docs/profiling.md): the jonoffcpu profiler, its requirements and options, the files of a profiled run
- and the measurement recording.
-- [Metrics](docs/metrics.md): the metrics stack, VictoriaMetrics and Grafana with the Pulsar dashboards, which collects
- the metrics of the brokers, the bookies and ZooKeeper during runs, and renders Grafana's panels as images.
-- [Heap dumps](docs/heap-dumps.md): heap dumps of the broker, the gateways and the applications when they run out of
- memory, at the highest heap usage and at given times.
+- [Comparing revisions](docs/comparing-revisions.md): running an A/B comparison, comparing with a released Pulsar,
+ and charting the comparison.
+- [Profiling](docs/profiling.md): async-profiler, JDK Flight Recorder and jonoffcpu, their requirements and options,
+ the files of a profiled run and the measurement recording.
- [Analyzing profiles](docs/analyzing-profiles.md): finding what to optimize, comparing profiles,
[flame graphs of other recordings](docs/analyzing-profiles.md#flame-graphs-of-other-recordings), such as those of
profiled tests, integration tests and benchmarks, the tools that read the recordings, including JDK Mission Control
and [AI agents](docs/analyzing-profiles.md#ai-agent-analysis), and
[analyzing heap dumps](docs/analyzing-profiles.md#heap-dumps-and-memory-leaks).
-- [Comparing revisions](docs/comparing-revisions.md): running an A/B comparison.
+- [Metrics](docs/metrics.md): the metrics stack, VictoriaMetrics and Grafana with the Pulsar dashboards, which collects
+ the metrics of the brokers, the bookies and ZooKeeper during runs, and renders Grafana's panels as images.
+- [Heap dumps](docs/heap-dumps.md): heap dumps of the broker, the gateways and the applications when they run out of
+ memory, at the highest heap usage and at given times.
- [The performance testing environment setup](environment/README.md): configuring a Linux host for consistent
- results, and freeing Docker disk space.
+ results, taking the disk out of the way of large-message runs, and freeing Docker disk space.
- [The legacy TestNG profiling runner](docs/legacy-testng-runner/README.md): the deprecated `pulsar-perf` based runner
and its scenarios.
@@ -459,6 +477,6 @@ details.
| [`tools`](tools) | The workload applications, which run in the workload containers |
| [`common`](common) | The scenario loader, shared by the launcher and the workload applications |
| [`metrics`](metrics) | The metrics stack, VictoriaMetrics and Grafana, which Docker Compose runs from its compose file, and the collection of a run's metrics |
-| [`report-tool`](report-tool) | Writes the run and profile reports, charts and flame graphs, and serves the reports over HTTP |
+| [`report-tool`](report-tool) | Writes the run and profile reports, charts and flame graphs, the comparison charts of two runs, and serves the reports over HTTP |
| [`environment`](environment/README.md) | Scripts that configure the host for performance testing and free Docker disk space |
| [`docs`](docs) | The reference documentation |
diff --git a/tests/performance/docs/analyzing-profiles.md b/tests/performance/docs/analyzing-profiles.md
index 1c0045c8814e..18cf55d439bb 100644
--- a/tests/performance/docs/analyzing-profiles.md
+++ b/tests/performance/docs/analyzing-profiles.md
@@ -221,7 +221,7 @@ ORDER BY samples DESC
LIMIT 45;
```
-### Analysing collapsed stacktrace files with quack_flamegraph
+### Analyzing collapsed stacktrace files with quack_flamegraph
AI agents can automate analysis by combining quack_flamegraph's functions in DuckDB queries and views, as the
`leaf_by_frame` view above demonstrates. They can join results and add `WHERE` clauses to narrow the analysis,
diff --git a/tests/performance/docs/comparing-revisions.md b/tests/performance/docs/comparing-revisions.md
index 458e6b063801..a693d5a13ac3 100644
--- a/tests/performance/docs/comparing-revisions.md
+++ b/tests/performance/docs/comparing-revisions.md
@@ -33,17 +33,18 @@ use the CPU, and keep the disk that holds Docker's data less than 90 % full.
## Check out both revisions
-Use a worktree for each revision, so that both can be built and run without switching branches. Both revisions need
-the same performance tests, and everything they build from, so that only the code under test differs.
+Use a worktree for each revision, so that both can be built and run without switching branches. Both revisions need the
+same version of the Pulsar Performance Testing Framework, and everything it builds from, so that only the code under
+test differs.
-When the candidate changes only the code under test, and the base revision already has the same performance tests,
-check out the base revision as the baseline. For example, from a candidate checkout based on `origin/master`:
+When the candidate changes only the code under test, and the base revision already has the same version of the
+framework, check out the base revision as the baseline. For example, from a candidate checkout based on `origin/master`:
```bash
git worktree add --detach ../pulsar-baseline origin/master
```
-Otherwise, for example when the candidate also changes the performance tests, or the base revision doesn't have them
+Otherwise, for example when the candidate also changes the framework, or the base revision doesn't have them
yet, create the baseline from the candidate and restore only the code under test from the base revision. Take the
paths from `git diff --stat origin/master...HEAD`, such as the broker's and the managed ledger's sources, and commit
the result, so that the baseline's runs name their own commit:
@@ -133,5 +134,34 @@ runs go under `pulsar-latest`, which gathers the runs of different releases as `
- Report percentage changes alongside the absolute values, and say which direction is better for each measure.
- Compare profiles with the correlator, see [Comparing two profiles](analyzing-profiles.md#comparing-two-profiles).
+### Comparison charts
+
+`compareRuns` charts a baseline run (A) against a comparison run (B), such as each revision's median run, with the
+same axes for both runs, so that their values can be compared by position:
+
+```bash
+./gradlew :tests:performance:report-tool:compareRuns --args="--baseline <A run directory> \
+ --comparison <B run directory> --baseline-label 4.0.13 --comparison-label master --output <directory>"
+```
+
+It writes the throughput (published and consumed), the backlog, and the publish and end-to-end latency by
+percentile, the latency both on a linear and on a logarithmic axis (`latency-percentiles-log`), for latencies that
+differ by orders of magnitude. Each chart comes in two forms:
+
+| File | Form |
+|---|---|
+| `<chart>-<A label>-vs-<B label>.svg` | Both runs in one diagram: A's lines blue, B's orange |
+| `<chart>-<A label>-vs-<B label>-separate.svg` | A above B, in two panels with the same axes, each tinted in its run's color and labeled with it |
+
+The labels in the file names, such as `throughput-4.0.13-vs-5.0.0.svg`, keep the charts of several comparisons apart
+in one directory; characters other than letters, digits, dots and underscores become underscores, and
+`--no-labels-in-file-names` leaves them out (`throughput.svg`).
+
+Within a run, the published rate and the publish latency are dashed, and the consumed rate and the end-to-end latency
+solid. Time runs from each run's measurement start, and markers in each run's color show when its gateways finished
+publishing and when its consumers received their last measured message; labels of close markers stack. A label defaults to the run's name, the name of its directory's parent; `--baseline-label` and `--comparison-label`
+set it, such as to a release version in place of a branch name. The charts are read from the run
+directories, so runs of different checkouts or releases, and runs made before this task, can be compared.
+
Each run's report records the commit it was made from, and whether the checkout had uncommitted changes. Keep the
scenario, any local changes, the runs and the conclusions together, so that a later run can reproduce the workload.
diff --git a/tests/performance/docs/metrics.md b/tests/performance/docs/metrics.md
index 088d8ed8ae36..841eb1ae8dd1 100644
--- a/tests/performance/docs/metrics.md
+++ b/tests/performance/docs/metrics.md
@@ -185,7 +185,7 @@ curl --get "$(jq -r .victoriaMetrics.prometheusApi metrics.json)query_range" \
--data-urlencode "end=$(jq -r '.endEpochMs / 1000' metrics.json)" --data-urlencode step=5
```
-Some panels stay empty, since they show what the performance tests don't run, such as proxies, functions and
+Some panels stay empty, since they show what the scenarios don't run, such as proxies, functions and
connectors, or metrics of Kubernetes and of the node exporter.
## Rendering panels as images
diff --git a/tests/performance/docs/run-reports.md b/tests/performance/docs/run-reports.md
index 93ee639e6b98..59a5a090d093 100644
--- a/tests/performance/docs/run-reports.md
+++ b/tests/performance/docs/run-reports.md
@@ -264,7 +264,7 @@ on a host with several interfaces, such as Docker's or a VPN's, when that addres
or when you reach the server through an SSH tunnel or a proxy. `serveReports` prints the same base URL when it starts.
A run written with `--output` outside the reports root has no URL.
-When the performance tests run on a separate machine, start the server there. It's reachable from your own machine
+When the framework runs on a separate machine, start the server there. It's reachable from your own machine
only when `performance.reportsServer.bindAddress` is set as above, in that machine's `~/.gradle/gradle.properties` or
with `-P`, or through an SSH tunnel with the default address. The tunnel forwards local ports to those loopback
addresses over the encrypted SSH connection. This one forwards the reports server, and Grafana and VictoriaMetrics of
diff --git a/tests/performance/environment/README.md b/tests/performance/environment/README.md
index 36bccc5fc637..1d929e912c16 100644
--- a/tests/performance/environment/README.md
+++ b/tests/performance/environment/README.md
@@ -21,11 +21,12 @@
# Performance testing environment
-Results of a run depend on the state of the host: a CPU running at turbo frequencies slows down as it heats up and
-throttles, and daemons that manage power change CPU and device settings during a run. The scripts in `scripts/`
-configure a Linux host for consistent results across runs with a [TuneD](https://tuned-project.org/) profile, and
-restore a configuration that allows power saving afterwards. The TuneD daemon only runs between `start` and `stop`. The
-same setup gives consistent results for [the JMH microbenchmarks](../../../microbench/README.md).
+Results of a run of the [Pulsar Performance Testing Framework](../README.md) depend on the state of the host: a CPU
+running at turbo frequencies slows down as it heats up and throttles, and daemons that manage power change CPU and
+device settings during a run. The scripts in `scripts/` configure a Linux host for consistent results across runs with
+a [TuneD](https://tuned-project.org/) profile, and restore a configuration that allows power saving afterwards. The
+TuneD daemon only runs between `start` and `stop`. The same setup gives consistent results for
+[the JMH microbenchmarks](../../../microbench/README.md).
`install` installs TuneD with `apt-get` when it is missing, on Debian based Linux distributions such as Debian,
Ubuntu and Pop!_OS; on other distributions, install TuneD first. It updates Docker's logging configuration with `jq`,
@@ -35,29 +36,12 @@ on any Linux distribution where TuneD and the profile are installed.
| Command | What it does |
|---|---|
| `install` | Installs TuneD when it is missing (Debian based distributions), disables TuneD's dynamic tuning, installs the `performance-testing` TuneD profile without activating it, limits the size of Docker's container logs and leaves the TuneD daemon disabled; when the profile is active, it applies the updated profile. Run once, and again after the profile changes. |
-| `start` | Checks that the host is on AC power and warns when Docker's disk is 90 % full, stops `thermald` (and `com.system76.PowerDaemon.service` on Pop!_OS), activates and verifies the `performance-testing` profile and skips the `:tests:integration:tuneKernelPerfEvents` task in `~/.gradle/gradle.properties`. |
-| `stop` | Switches TuneD to the `balanced` profile, stops TuneD, applies the system's configured dirty page limits and swappiness again, starts the stopped daemons again and removes the Gradle property. |
-| `validate` | Checks that the host is ready for performance tests, see [Checking the host](#checking-the-host). Runs on Linux and macOS. |
+| `start` | Checks that the host is on AC power and warns when Docker's disk is 90 % full, stops `thermald` (and `com.system76.PowerDaemon.service` on Pop!_OS), activates and verifies the `performance-testing` profile and skips the `:tests:integration:tuneKernelPerfEvents` task in `~/.gradle/gradle.properties`. With `--disable-write-barriers`, it also disables the write barriers of the file system that holds the containers' file systems, see [Disabling write barriers](#disabling-write-ba [...]
+| `stop` | Switches TuneD to the `balanced` profile, stops TuneD, applies the system's configured dirty page limits and swappiness again, starts the stopped daemons again, enables the write barriers that `start` disabled and removes the Gradle property. |
+| `validate` | Checks that the host is ready for performance testing, see [Checking the host](#checking-the-host). Runs on Linux and macOS. |
`install`, `start` and `stop` run as root: `sudo scripts/configure-perf-test-environment.sh start`.
-## What the profile changes
-
-The `performance-testing` profile includes TuneD's `latency-performance` profile (performance CPU governor, CPU idle
-states limited to C1, `min_perf_pct=100`) and adds:
-
-- **Turbo disabled**, so that the CPU runs at a fixed base frequency. Turbo frequencies depend on the temperature and
- the power budget of the CPU, which vary between runs and within a run. Absolute throughput is lower than with
- turbo, but comparisons between runs are more reliable, and cooling down between runs matters much less.
-- `vm.swappiness=1` and NUMA balancing disabled. The host's own dirty page limits are kept, and `stop` applies the
- system's configured dirty page limits and swappiness again after TuneD has switched back.
-- The `none` I/O scheduler, the `performance` ACPI platform profile (fans and power limits of laptops) and NVMe
- Autonomous Power State Transitions disabled.
-- Settings for profiling and for `-XX:+UseTransparentHugePages`: the perf event and BPF limits, the NMI watchdog
- disabled and Transparent Huge Pages in `madvise` mode with `defrag=madvise`.
-
-`stop` restores the previous values, except the profiling and Transparent Huge Pages settings, which stay in place.
-
## Setup
Install the profile from this directory:
@@ -91,6 +75,16 @@ visudo -cf "$tmp" && sudo install -o root -g root -m 0440 "$tmp" /etc/sudoers.d/
rm "$tmp"
```
+A command with arguments in a sudoers rule allows exactly those arguments, so the rule above doesn't allow
+`start --disable-write-barriers`, see [Disabling write barriers](#disabling-write-barriers). To allow it, set `rule`
+to this instead and run the other commands above again, which replace the rule:
+
+```sh
+rule="$USER ALL=(root) NOPASSWD: /usr/local/sbin/configure-perf-test-environment.sh start, \
+/usr/local/sbin/configure-perf-test-environment.sh start --disable-write-barriers, \
+/usr/local/sbin/configure-perf-test-environment.sh stop"
+```
+
The rule takes effect immediately. Check it:
```sh
@@ -131,17 +125,18 @@ sudo /usr/local/sbin/configure-perf-test-environment.sh stop
```
Without the sudoers rule, run `tests/performance/environment/scripts/configure-perf-test-environment.sh` instead.
-See [the performance testing guide](../README.md) for the scenarios and the `profile` task.
+See [the framework's guide](../README.md) for the scenarios and the `profile` task.
Before a run:
- Keep the disk that holds Docker's data less than 90 % full. BookKeeper bookies switch to read-only mode when the
- disk is 95 % full. `scripts/docker-cleanup.sh` frees space, see below.
+ disk is 95 % full. `scripts/docker-cleanup.sh` frees space, see
+ [Freeing Docker disk space](#freeing-docker-disk-space).
- Close applications that use the CPU, such as browsers and IDEs.
## Checking the host
-`validate` checks that the host is ready for performance tests, for example in a script or by an AI agent before a
+`validate` checks that the host is ready for performance testing, for example in a script or by an AI agent before a
series of runs. It runs on Linux and on macOS:
```sh
@@ -174,6 +169,51 @@ bits 1 and 3 set, 2 + 8: Docker's disk is too full and the host isn't configured
A failed setting check while the profile is active means that the installed profile is older than the script: run
`install` again, then `start`.
+## What the profile changes
+
+The `performance-testing` profile includes TuneD's `latency-performance` profile (performance CPU governor, CPU idle
+states limited to C1, `min_perf_pct=100`) and adds:
+
+- **Turbo disabled**, so that the CPU runs at a fixed base frequency. Turbo frequencies depend on the temperature and
+ the power budget of the CPU, which vary between runs and within a run. Absolute throughput is lower than with
+ turbo, but comparisons between runs are more reliable, and cooling down between runs matters much less.
+- `vm.swappiness=1` and NUMA balancing disabled. The host's own dirty page limits are kept, and `stop` applies the
+ system's configured dirty page limits and swappiness again after TuneD has switched back.
+- The `none` I/O scheduler, the `performance` ACPI platform profile (fans and power limits of laptops) and NVMe
+ Autonomous Power State Transitions disabled.
+- Settings for profiling and for `-XX:+UseTransparentHugePages`: the perf event and BPF limits, the NMI watchdog
+ disabled and Transparent Huge Pages in `madvise` mode with `defrag=madvise`.
+
+`stop` restores the previous values, except the profiling and Transparent Huge Pages settings, which stay in place.
+
+## Disabling write barriers
+
+The bookies' ledgers are in their containers' file systems, on the host's disk. Each time a bookie flushes its write
+cache, it syncs its entry logs and its RocksDB index (`fdatasync`, `fsync`), and the file system then makes the disk
+write its volatile cache too. A server's disk with power loss protection acknowledges that at once; most laptop and
+desktop disks don't, so with large unbatched entries the syncs can make the disk the limit of a run, which a production
+cluster wouldn't have: the bookies' flushes take long (`bookie_flush`), they throttle writes
+(`bookie_throttled_write`), and the broker's adds wait while no thread is busy. The bookies' journal isn't synced in
+the test cluster, and with [`journalTmpfs`](../scenarios/README.md#the-bookies-journal-on-a-tmpfs) it isn't on the
+disk at all; try that first, since it has no risk.
+
+`start --disable-write-barriers` remounts the file system that holds the containers' file systems without write
+barriers (`barrier=0` on ext4, `nobarrier` on btrfs; other file systems are left as they are), so that `fsync` no
+longer waits for the disk's cache. It finds that file system from the overlay of a disposable container: it is the
+one of Docker's data directory with Docker's own storage drivers, and of containerd's, such as `/var/lib/containerd`,
+with the containerd image store. When the host has several file systems, only that one is remounted.
+`stop`, or a `start` without the option, turns them on again, and so does a reboot.
+
+**Use it only when losing the file system's data is acceptable.** Without write barriers, the disk can write the file
+system's journal out of order. If the disk then loses its cache, in a power loss or when the host is powered off
+without shutting down, the file system can be corrupted, and with it any file on it, not only Docker's. On a host
+whose Docker data is on the root file system, that is the whole system. A crash of the kernel is less of a risk, since
+the disk keeps its power and writes its cache, but a hard reset of the disk can lose it too.
+
+```sh
+sudo scripts/configure-perf-test-environment.sh start --disable-write-barriers
+```
+
## Freeing Docker disk space
Every `-Pdocker.tag` leaves Pulsar images of about 750 MB behind, and Docker's pruning doesn't remove tagged images.
diff --git a/tests/performance/environment/scripts/configure-perf-test-environment.sh b/tests/performance/environment/scripts/configure-perf-test-environment.sh
index 5cb57b11c532..5f76240f5103 100755
--- a/tests/performance/environment/scripts/configure-perf-test-environment.sh
+++ b/tests/performance/environment/scripts/configure-perf-test-environment.sh
@@ -26,9 +26,12 @@
# start checks that the host is on AC power and has disk space, stops daemons that
# would throttle or retune the host, activates the performance-testing TuneD
# profile and skips the Gradle task that applies the same kernel settings in
-# ~/.gradle/gradle.properties of the user running sudo
+# ~/.gradle/gradle.properties of the user running sudo. With
+# --disable-write-barriers, it also remounts the file system of the containers'
+# file systems without write barriers, see disable_write_barriers
# stop switches TuneD to a balanced profile that allows power saving, stops TuneD,
-# starts the daemons stopped by "start" again and removes the Gradle property
+# starts the daemons stopped by "start" again, enables the write barriers that
+# "start" disabled and removes the Gradle property
# validate checks that the host is ready for performance tests: on every operating system
# that Docker is available and has disk space, and on Linux also AC power, the
# active TuneD profile and the settings it applies. It prints each check to stdout
@@ -69,9 +72,12 @@ EXIT_HOST_CONFIGURATION=8
GRADLE_SKIP_PROPERTY="inttest.asyncprofiler.skipPerfEventTuning"
GRADLE_PROPERTIES_BEGIN="# BEGIN added by configure-perf-test-environment.sh start, removed by stop"
GRADLE_PROPERTIES_END="# END added by configure-perf-test-environment.sh start"
+# The mount point whose write barriers "start --disable-write-barriers" disabled, for "stop". /run is
+# emptied at boot, when the file system is mounted with its own options again.
+WRITE_BARRIERS_STATE="/run/configure-perf-test-environment/write-barriers-disabled"
usage() {
- echo "Usage: sudo $0 install|start|stop, or $0 validate" >&2
+ echo "Usage: sudo $0 install|start [--disable-write-barriers]|stop, or $0 validate" >&2
}
require_root() {
@@ -367,6 +373,96 @@ print_perf_settings() {
echo
}
+# The directory of a container's writable layer, where the bookies' ledgers are: the upper directory
+# of the overlay mount of a disposable container's root. It is in Docker's data directory with Docker's
+# own storage drivers, and in containerd's, such as /var/lib/containerd, with the containerd image store.
+container_layer_directory() {
+ local id pid upper
+ id="$(docker run --detach --rm "${DISK_CHECK_IMAGE}" sleep 60 2>/dev/null)" || return 1
+ pid="$(docker inspect --format '{{.State.Pid}}' "${id}" 2>/dev/null || true)"
+ # The root's line in mountinfo ends with the overlay's options, which have upperdir=
+ upper="$(awk '$5 == "/" { print $NF }' "/proc/${pid:-0}/mountinfo" 2>/dev/null \
+ | tr ',' '\n' | sed -n 's/^upperdir=//p' || true)"
+ docker rm --force "${id}" >/dev/null 2>&1 || true
+ [[ -n "${upper}" ]] && echo "${upper}"
+}
+
+# The mount point and the type of the file system of the containers' writable layers, where the
+# bookies' ledgers are. Only this file system is changed, when the host has several.
+docker_data_mount() {
+ local directory
+ if ! directory="$(container_layer_directory)"; then
+ # Docker's data directory, when a container's layer can't be found, such as with a storage
+ # driver that doesn't use overlays
+ if ! directory="$(docker info --format '{{.DockerRootDir}}' 2>/dev/null)" || [[ -z "${directory}" ]]; then
+ echo "ERROR: Can't find where Docker keeps the containers' file systems. Start Docker first." >&2
+ return 1
+ fi
+ echo "The containers' layers weren't found, using Docker's data directory ${directory}" >&2
+ fi
+ # The container's layer has been removed with the container, and its parent directory stays
+ while [[ ! -e "${directory}" && "${directory}" != / ]]; do
+ directory="$(dirname "${directory}")"
+ done
+ findmnt --noheadings --output TARGET,FSTYPE --target "${directory}"
+}
+
+# The mount options that turn the write barriers of a file system type off and on, or nothing for a
+# type that doesn't have them. XFS no longer has an option to turn them off.
+write_barrier_options() {
+ case "$1" in
+ ext4) echo "barrier=0 barrier=1" ;;
+ btrfs) echo "nobarrier barrier" ;;
+ *) return 1 ;;
+ esac
+}
+
+write_barriers_disabled() {
+ [[ ",$(findmnt --noheadings --output OPTIONS --mountpoint "$1")," =~ ,(nobarrier|barrier=0), ]]
+}
+
+# Remounts the file system of the containers' file systems without write barriers: an fsync then no longer waits for
+# the disk to write its volatile cache, which BookKeeper's ledger storage does at each flush. The disk
+# may then write the file system's journal out of order, so losing its cache, in a power loss or when
+# the host is powered off without shutting down, can corrupt the file system and any file on it, not
+# only Docker's. "stop" turns them on again.
+disable_write_barriers() {
+ local target fstype options
+ read -r target fstype < <(docker_data_mount) || exit 1
+ if ! options="$(write_barrier_options "${fstype}")"; then
+ echo "WARNING: The containers' file systems are on ${target}, which is ${fstype}, whose write barriers" \
+ "can't be disabled; it's left as it is." >&2
+ return
+ fi
+ if write_barriers_disabled "${target}"; then
+ echo "The write barriers of ${target} are already disabled"
+ return
+ fi
+ echo "Disabling the write barriers of ${target} (${fstype}), where the containers' file systems are"
+ echo "WARNING: Until \"$0 stop\", a power loss or powering the host off without shutting it down can" \
+ "corrupt the file system of ${target} and any file on it." >&2
+ mount -o "remount,${options%% *}" "${target}"
+ mkdir -p "$(dirname "${WRITE_BARRIERS_STATE}")"
+ echo "${target} ${fstype}" >"${WRITE_BARRIERS_STATE}"
+ if ! write_barriers_disabled "${target}"; then
+ echo "ERROR: The write barriers of ${target} are still enabled." >&2
+ exit 1
+ fi
+}
+
+# Enables the write barriers that "start --disable-write-barriers" disabled
+restore_write_barriers() {
+ local target fstype options
+ if [[ ! -f "${WRITE_BARRIERS_STATE}" ]]; then
+ return
+ fi
+ read -r target fstype <"${WRITE_BARRIERS_STATE}"
+ options="$(write_barrier_options "${fstype}")"
+ echo "Enabling the write barriers of ${target} again"
+ mount -o "remount,${options##* }" "${target}"
+ rm -f "${WRITE_BARRIERS_STATE}"
+}
+
# The Docker logging configuration is updated with jq, which is expected on the host. Checked before
# install changes anything
check_jq_for_docker_logging() {
@@ -548,12 +644,30 @@ activate_perf_profile() {
}
start() {
+ local disable_barriers=false
+ while (($# > 0)); do
+ case "$1" in
+ --disable-write-barriers) disable_barriers=true ;;
+ *)
+ usage
+ exit 1
+ ;;
+ esac
+ shift
+ done
+
check_perf_profile_installed
check_ac_power
check_disk_space
stop_thermald
stop_distro_specific_services
activate_perf_profile
+ if [[ "${disable_barriers}" == true ]]; then
+ disable_write_barriers
+ else
+ # A previous "start --disable-write-barriers" disabled them
+ restore_write_barriers
+ fi
add_gradle_properties
echo
@@ -613,6 +727,7 @@ stop() {
restore_system_memory_settings
start_distro_specific_services
start_thermald
+ restore_write_barriers
remove_gradle_properties
echo
@@ -787,7 +902,7 @@ case "${1:-}" in
;;
start)
require_root start
- start
+ start "${@:2}"
;;
stop)
require_root stop
diff --git a/tests/performance/launcher/src/main/java/org/apache/pulsar/tests/performance/launcher/ClusterSettings.java b/tests/performance/launcher/src/main/java/org/apache/pulsar/tests/performance/launcher/ClusterSettings.java
index d84cd9a4cf9a..345101211744 100644
--- a/tests/performance/launcher/src/main/java/org/apache/pulsar/tests/performance/launcher/ClusterSettings.java
+++ b/tests/performance/launcher/src/main/java/org/apache/pulsar/tests/performance/launcher/ClusterSettings.java
@@ -24,11 +24,12 @@ import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
-import java.util.Set;
+import java.util.regex.Pattern;
/**
- * A scenario's {@code cluster} section: the replicas and the environment of the brokers and of the bookies. The
- * workload's containers have their environment in the workload's section.
+ * A scenario's {@code cluster} section: the replicas and the environment of the brokers and of the bookies, and
+ * whether the bookies' journals are on a tmpfs. The workload's containers have their environment in the workload's
+ * section.
*
* <pre>
* cluster:
@@ -40,6 +41,7 @@ import java.util.Set;
* replicas: 3
* env:
* journalMaxSizeMB: "8192"
+ * journalTmpfs: 2g
* </pre>
*/
record ClusterSettings(Component brokers, Component bookies) {
@@ -48,15 +50,57 @@ record ClusterSettings(Component brokers, Component bookies) {
private static final List<String> KEYS = List.of(BROKERS, BOOKIES);
private static final String REPLICAS = "replicas";
private static final String ENV = "env";
- private static final Set<String> COMPONENT_KEYS = Set.of(REPLICAS, ENV);
+ static final String JOURNAL_TMPFS = "journalTmpfs";
+ private static final List<String> BROKER_KEYS = List.of(REPLICAS, ENV);
+ private static final List<String> BOOKIE_KEYS = List.of(REPLICAS, ENV, JOURNAL_TMPFS);
+ /** A tmpfs size as Docker takes it: bytes, or a number with the unit k, m or g. */
+ private static final Pattern TMPFS_SIZE = Pattern.compile("[1-9][0-9]*[kmgKMG]?");
+ /**
+ * The bookies' journal directory on the tmpfs, in place of the cluster's directory on the disk. Docker creates the
+ * missing parent directories of a mount as root, so it is in /pulsar/data, which the image has and the bookie's
+ * user owns: in data/bookkeeper, the bookie couldn't create its ledger directories.
+ */
+ static final String TMPFS_JOURNAL_DIRECTORY = "/pulsar/data/journal-tmpfs";
+ private static final String JOURNAL_DIRECTORY = "journalDirectory";
+ private static final String JOURNAL_DIRECTORIES = "journalDirectories";
+ /**
+ * The journal settings of a journal on a tmpfs, in place of the scenario's. A bookie deletes its old journal files
+ * when it has flushed its write cache, but it keeps the current file until the file is full, and the backups.
+ * Files of 256 MB without backups keep each bookie's journal to about 512 MB of memory.
+ */
+ static final Map<String, String> TMPFS_JOURNAL_SETTINGS = Map.of(
+ "journalMaxSizeMB", "256",
+ "journalMaxBackups", "0");
/**
* A component of the cluster.
*
* @param replicas how many containers run it
* @param env the environment variables of its containers, such as Pulsar settings and {@code PULSAR_MEM}
+ * @param journalTmpfs the size of the tmpfs that holds each bookie's journal, such as {@code 2g}, or null when
+ * the journal is on the disk; bookies only
*/
- record Component(int replicas, Map<String, String> env) {
+ record Component(int replicas, Map<String, String> env, String journalTmpfs) {
+ /**
+ * The environment of the containers: {@link #env()}, and with a journal on a tmpfs the journal's directory
+ * and {@link #TMPFS_JOURNAL_SETTINGS}, which replace those of {@link #env()}.
+ */
+ Map<String, String> containerEnv() {
+ if (journalTmpfs == null) {
+ return env;
+ }
+ Map<String, String> containerEnv = new LinkedHashMap<>(env);
+ containerEnv.putAll(TMPFS_JOURNAL_SETTINGS);
+ containerEnv.put(JOURNAL_DIRECTORY, TMPFS_JOURNAL_DIRECTORY);
+ return containerEnv;
+ }
+
+ /** The tmpfs mount of a bookie's journal, for Testcontainers' {@code withTmpFs}; empty without one. */
+ Map<String, String> journalTmpfsMount() {
+ // The bookie doesn't run as root, so the tmpfs is writable by every user, as /tmp is
+ return journalTmpfs == null ? Map.of()
+ : Map.of(TMPFS_JOURNAL_DIRECTORY, "rw,size=" + journalTmpfs + ",mode=1777");
+ }
}
/** Reads the {@code cluster} section, which needs the brokers' and the bookies' replicas. */
@@ -68,22 +112,23 @@ record ClusterSettings(Component brokers, Component bookies) {
cluster.fieldNames().forEachRemaining(field -> {
if (!KEYS.contains(field)) {
throw new IllegalArgumentException("cluster." + field + " isn't a setting; cluster has " + KEYS
- + ", and brokers and bookies each have " + REPLICAS + " and " + ENV);
+ + ", brokers have " + BROKER_KEYS + " and bookies " + BOOKIE_KEYS);
}
});
- return new ClusterSettings(component(mapper, cluster, BROKERS), component(mapper, cluster, BOOKIES));
+ return new ClusterSettings(component(mapper, cluster, BROKERS, BROKER_KEYS),
+ component(mapper, cluster, BOOKIES, BOOKIE_KEYS));
}
- private static Component component(ObjectMapper mapper, JsonNode cluster, String name) {
+ private static Component component(ObjectMapper mapper, JsonNode cluster, String name, List<String> keys) {
JsonNode section = cluster.path(name);
if (!section.isObject()) {
throw new IllegalArgumentException("cluster." + name + " must be a mapping with " + REPLICAS + " and "
+ ENV);
}
section.fieldNames().forEachRemaining(field -> {
- if (!COMPONENT_KEYS.contains(field)) {
+ if (!keys.contains(field)) {
throw new IllegalArgumentException("cluster." + name + "." + field + " isn't a setting; cluster."
- + name + " has " + REPLICAS + " and " + ENV);
+ + name + " has " + keys);
}
});
JsonNode replicas = section.path(REPLICAS);
@@ -91,7 +136,28 @@ record ClusterSettings(Component brokers, Component bookies) {
throw new IllegalArgumentException("cluster." + name + "." + REPLICAS + " must be a positive number, not "
+ (replicas.isMissingNode() ? "missing" : replicas.toString()));
}
- return new Component(replicas.intValue(), env(mapper, section.path(ENV), "cluster." + name + "." + ENV));
+ Map<String, String> env = env(mapper, section.path(ENV), "cluster." + name + "." + ENV);
+ return new Component(replicas.intValue(), env, journalTmpfs(section.path(JOURNAL_TMPFS), env, name));
+ }
+
+ /** The size of the bookies' journal tmpfs, null when the setting is missing or null. */
+ private static String journalTmpfs(JsonNode setting, Map<String, String> env, String name) {
+ if (setting.isMissingNode() || setting.isNull()) {
+ return null;
+ }
+ String path = "cluster." + name + "." + JOURNAL_TMPFS;
+ String size = setting.asText();
+ if (!setting.isValueNode() || !TMPFS_SIZE.matcher(size).matches()) {
+ throw new IllegalArgumentException(path + " must be the size of the tmpfs, such as 2g, 512m or a number"
+ + " of bytes, not " + setting);
+ }
+ for (String directory : List.of(JOURNAL_DIRECTORY, JOURNAL_DIRECTORIES)) {
+ if (env.containsKey(directory)) {
+ throw new IllegalArgumentException(path + " puts the journal in " + TMPFS_JOURNAL_DIRECTORY
+ + ", so cluster." + name + ".env can't set " + directory);
+ }
+ }
+ return size;
}
/** An {@code env} mapping of environment variables, empty when missing; {@code path} names it in errors. */
diff --git a/tests/performance/launcher/src/main/java/org/apache/pulsar/tests/performance/launcher/PerformanceLauncher.java b/tests/performance/launcher/src/main/java/org/apache/pulsar/tests/performance/launcher/PerformanceLauncher.java
index 3cc68e9f583f..35b419f6f785 100644
--- a/tests/performance/launcher/src/main/java/org/apache/pulsar/tests/performance/launcher/PerformanceLauncher.java
+++ b/tests/performance/launcher/src/main/java/org/apache/pulsar/tests/performance/launcher/PerformanceLauncher.java
@@ -380,7 +380,7 @@ public class PerformanceLauncher implements Callable<Integer> {
.clusterImage(clusterImage)
.brokerEnvs(brokerEnv)
.brokerMountFiles(brokerMounts)
- .bookkeeperEnvs(metricsSettings.withBookieStatsSettings(clusterSettings.bookies().env()))
+ .bookkeeperEnvs(metricsSettings.withBookieStatsSettings(clusterSettings.bookies().containerEnv()))
.build();
HostStatsSampler.Sensors sensors = HostStatsSampler.discover(sysfs);
@@ -391,6 +391,11 @@ public class PerformanceLauncher implements Callable<Integer> {
}
Thread measurementGate = null;
PulsarCluster cluster = PulsarCluster.forSpec(spec);
+ // The containers are created, not started yet
+ Map<String, String> journalTmpfsMount = clusterSettings.bookies().journalTmpfsMount();
+ if (!journalTmpfsMount.isEmpty()) {
+ cluster.getBookies().forEach(bookie -> bookie.withTmpFs(journalTmpfsMount));
+ }
Path applicationsOutput = runOutput.resolve(APPLICATIONS_DIRECTORY);
GenericContainer<?> consumer = null;
GenericContainer<?> producer = null;
@@ -412,8 +417,10 @@ public class PerformanceLauncher implements Callable<Integer> {
ReportsUrl.DEFAULT_BIND_ADDRESS);
ZonedDateTime workloadFinished;
try {
- status(String.format(Locale.ROOT, "Starting the Pulsar cluster: %d broker(s), %d bookie(s)",
- spec.numBrokers(), spec.numBookies()));
+ String journalTmpfs = clusterSettings.bookies().journalTmpfs();
+ status(String.format(Locale.ROOT, "Starting the Pulsar cluster: %d broker(s), %d bookie(s)%s",
+ spec.numBrokers(), spec.numBookies(),
+ journalTmpfs != null ? ", with each bookie's journal on a tmpfs of " + journalTmpfs : ""));
long clusterStart = System.nanoTime();
cluster.start();
status(String.format(Locale.ROOT, "Started the Pulsar cluster in %.0f s",
diff --git a/tests/performance/launcher/src/test/java/org/apache/pulsar/tests/performance/launcher/ClusterSettingsTest.java b/tests/performance/launcher/src/test/java/org/apache/pulsar/tests/performance/launcher/ClusterSettingsTest.java
index b7474a661d95..8a0ba733ffa0 100644
--- a/tests/performance/launcher/src/test/java/org/apache/pulsar/tests/performance/launcher/ClusterSettingsTest.java
+++ b/tests/performance/launcher/src/test/java/org/apache/pulsar/tests/performance/launcher/ClusterSettingsTest.java
@@ -20,6 +20,7 @@ package org.apache.pulsar.tests.performance.launcher;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.assertj.core.api.Assertions.entry;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
import org.testng.annotations.Test;
@@ -44,6 +45,50 @@ public class ClusterSettingsTest {
.containsEntry("managedLedgerMaxEntriesPerLedger", "100000000");
assertThat(settings.bookies().replicas()).isEqualTo(3);
assertThat(settings.bookies().env()).isEmpty();
+ assertThat(settings.bookies().journalTmpfs()).isNull();
+ assertThat(settings.bookies().containerEnv()).isEmpty();
+ assertThat(settings.bookies().journalTmpfsMount()).isEmpty();
+ }
+
+ @Test
+ public void putsTheBookiesJournalsOnATmpfs() throws Exception {
+ ClusterSettings settings = ClusterSettings.read(mapper, mapper.readTree("""
+ brokers:
+ replicas: 1
+ bookies:
+ replicas: 3
+ env:
+ journalMaxSizeMB: "8192"
+ gcWaitTime: "86400000"
+ journalTmpfs: 2g
+ """));
+
+ assertThat(settings.bookies().journalTmpfs()).isEqualTo("2g");
+ assertThat(settings.bookies().journalTmpfsMount()).containsExactly(
+ entry(ClusterSettings.TMPFS_JOURNAL_DIRECTORY, "rw,size=2g,mode=1777"));
+ // The tmpfs journal's settings replace the scenario's
+ assertThat(settings.bookies().containerEnv()).containsEntry("journalMaxSizeMB", "256")
+ .containsEntry("journalMaxBackups", "0")
+ .containsEntry("journalDirectory", ClusterSettings.TMPFS_JOURNAL_DIRECTORY)
+ .containsEntry("gcWaitTime", "86400000");
+ assertThat(settings.bookies().env()).containsEntry("journalMaxSizeMB", "8192");
+ }
+
+ @Test
+ public void rejectsAnInvalidJournalTmpfs() throws Exception {
+ assertThatThrownBy(() -> ClusterSettings.read(mapper, mapper.readTree("brokers: {replicas: 1}\n"
+ + "bookies: {replicas: 3, journalTmpfs: 2 GB}"))).isInstanceOf(IllegalArgumentException.class)
+ .hasMessageContaining("cluster.bookies.journalTmpfs must be the size of the tmpfs");
+ assertThatThrownBy(() -> ClusterSettings.read(mapper, mapper.readTree("brokers: {replicas: 1}\n"
+ + "bookies: {replicas: 3, journalTmpfs: [2g]}"))).isInstanceOf(IllegalArgumentException.class)
+ .hasMessageContaining("cluster.bookies.journalTmpfs must be the size of the tmpfs");
+ assertThatThrownBy(() -> ClusterSettings.read(mapper, mapper.readTree("brokers: {replicas: 1}\n"
+ + "bookies: {replicas: 3, journalTmpfs: 2g, env: {journalDirectories: /journal}}")))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessageContaining("cluster.bookies.env can't set journalDirectories");
+ assertThatThrownBy(() -> ClusterSettings.read(mapper, mapper.readTree("brokers: {replicas: 1, "
+ + "journalTmpfs: 2g}\nbookies: {replicas: 3}"))).isInstanceOf(IllegalArgumentException.class)
+ .hasMessageContaining("cluster.brokers.journalTmpfs isn't a setting");
}
@Test
diff --git a/tests/performance/report-tool/build.gradle.kts b/tests/performance/report-tool/build.gradle.kts
index 088f2b3c3b56..8911c641e74b 100644
--- a/tests/performance/report-tool/build.gradle.kts
+++ b/tests/performance/report-tool/build.gradle.kts
@@ -89,6 +89,18 @@ tasks.register<JavaExec>("renderHdrHistograms") {
mainClass.set("org.apache.pulsar.tests.performance.report.HdrHistogramRenderer")
}
+// Charts a baseline run (A) against a comparison run (B) with the same axes for both, combined in one diagram and in
+// separate panels: --args='--baseline <run directory> --comparison <run directory> --output <directory>', and
+// optionally --baseline-label, --comparison-label and --no-labels-in-file-names. Paths resolve from the repository
+// root.
+tasks.register<JavaExec>("compareRuns") {
+ group = "verification"
+ description = "Chart a baseline run (A) against a comparison run (B) with the same axes for both"
+ classpath = sourceSets.main.get().runtimeClasspath
+ mainClass.set("org.apache.pulsar.tests.performance.report.ComparisonCharts")
+ workingDir(rootProject.projectDir)
+}
+
// Serves the reports root over HTTP. -Pperformance.reportsDir chooses the root as for the launcher's tasks,
// -Pperformance.reportsServer.bindAddress and -Pperformance.reportsServer.port where to listen, and
// -Pperformance.reportsServer.baseUrl the URL that it prints, which the launcher's tasks print the reports' URLs with;
diff --git a/tests/performance/report-tool/src/main/java/org/apache/pulsar/tests/performance/report/ComparisonCharts.java b/tests/performance/report-tool/src/main/java/org/apache/pulsar/tests/performance/report/ComparisonCharts.java
new file mode 100644
index 000000000000..fa9da0d02e40
--- /dev/null
+++ b/tests/performance/report-tool/src/main/java/org/apache/pulsar/tests/performance/report/ComparisonCharts.java
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.tests.performance.report;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.MissingNode;
+import com.fasterxml.jackson.dataformat.yaml.YAMLMapper;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import org.HdrHistogram.Histogram;
+import org.HdrHistogram.HistogramIterationValue;
+import org.apache.pulsar.tests.performance.report.ComparisonRenderer.Chart;
+import org.apache.pulsar.tests.performance.report.ComparisonRenderer.Event;
+import org.apache.pulsar.tests.performance.report.ComparisonRenderer.Line;
+import org.apache.pulsar.tests.performance.report.ComparisonRenderer.Marker;
+import org.apache.pulsar.tests.performance.report.ComparisonRenderer.Side;
+import org.apache.pulsar.tests.performance.report.ComparisonRenderer.XAxis;
+import picocli.CommandLine;
+import picocli.CommandLine.Command;
+import picocli.CommandLine.Option;
+
+/**
+ * Compares two finished IoT runs, a baseline (A) and a comparison (B), in charts with the same axes for both: the
+ * throughput (published and consumed), the backlog, and the publish and end-to-end latency by percentile, on a
+ * linear and on a logarithmic axis, since latencies that differ by orders of magnitude flatten on a linear one. Each
+ * chart is written twice: {@code <chart>-<baselineLabel>-vs-<comparisonLabel>.svg} with both runs in one diagram,
+ * and {@code <chart>-<baselineLabel>-vs-<comparisonLabel>-separate.svg} with A above B in panels of their own, unless
+ * {@code --no-labels-in-file-names} leaves the labels out. The charts are read from the run
+ * directories' files: {@code topic-stats.csv}, {@code gateways/gateways-summary.json}, the applications' summaries and
+ * the HDR latency logs.
+ */
+@Command(name = "compare-runs", mixinStandardHelpOptions = true,
+ description = "Chart a baseline run (A) against a comparison run (B) with the same axes for both")
+public final class ComparisonCharts implements Callable<Integer> {
+ static final String THROUGHPUT = "throughput";
+ static final String BACKLOG = "backlog";
+ static final String LATENCY = "latency-percentiles";
+ static final String LATENCY_LOG = "latency-percentiles-log";
+ static final String SEPARATE_SUFFIX = "-separate";
+ // The percentile axis ends at six nines, as the run report's latency chart does
+ private static final double MAX_PERCENTILE = 99.9999;
+ private static final int PERCENTILE_TICKS_PER_HALF_DISTANCE = 5;
+ // The lowest value of the logarithmic latency axis, in milliseconds
+ private static final double LOG_LATENCY_MIN = 0.1;
+
+ @Option(names = "--baseline", required = true, description = "The baseline (A) run directory")
+ private Path baseline;
+
+ @Option(names = "--comparison", required = true, description = "The comparison (B) run directory")
+ private Path comparison;
+
+ @Option(names = "--baseline-label", description = "A's name in the charts; defaults to its run's name")
+ private String baselineLabel;
+
+ @Option(names = "--comparison-label", description = "B's name in the charts; defaults to its run's name")
+ private String comparisonLabel;
+
+ @Option(names = "--output", required = true, description = "The directory to write the charts to")
+ private Path output;
+
+ @Option(names = "--labels-in-file-names", negatable = true, defaultValue = "true", fallbackValue = "true",
+ description = "Name the charts <chart>-<A label>-vs-<B label>[-separate].svg; with"
+ + " --no-labels-in-file-names, <chart>[-separate].svg. Default: ${DEFAULT-VALUE}")
+ private boolean labelsInFileNames;
+
+ /**
+ * A run's data, with times in seconds since its measurement start.
+ *
+ * @param gatewaysFinishedSeconds when the gateways finished publishing
+ * @param consumersFinishedSeconds when the last application received its last measured message, {@link Double#NaN}
+ * when unknown
+ * @param consumed the rate that the subscriptions' consumers received messages at, summed over the subscriptions
+ * @param latencies the publish latency first, then each application's end-to-end latency, by percentile
+ */
+ record RunData(String label, String footer, double gatewaysFinishedSeconds, double consumersFinishedSeconds,
+ double[] seconds, double[] published, double[] consumed, double[] backlog, List<Curve> latencies) {
+ }
+
+ /** A named line of a latency chart: percentile axis positions and latencies in milliseconds. */
+ record Curve(String name, double[] positions, double[] millis) {
+ }
+
+ private ComparisonCharts() {
+ }
+
+ public static void main(String[] args) {
+ System.exit(new CommandLine(new ComparisonCharts()).execute(args));
+ }
+
+ @Override
+ public Integer call() throws Exception {
+ RunData a = read(baseline, baselineLabel);
+ RunData b = read(comparison, comparisonLabel);
+ for (Path chart : render(a, b, output, labelsInFileNames)) {
+ System.out.println(chart);
+ }
+ return 0;
+ }
+
+ /**
+ * Reads a run directory.
+ *
+ * @param label the run's name in the charts, or {@code null} for the run's name, its directory's parent
+ */
+ static RunData read(Path runDirectory, String label) throws IOException {
+ Path run = runDirectory.toAbsolutePath().normalize();
+ JsonNode producer = new ObjectMapper().readTree(run.resolve("gateways/gateways-summary.json").toFile());
+ long start = producer.path("measurementStartEpochMs").asLong();
+ double finished = (producer.path("measurementEndEpochMs").asLong() - start) / 1000.0;
+ RunReport.Samples samples = RunReport.readSamples(run.resolve(RunReport.TOPIC_STATS_FILE));
+ double[] seconds = new double[samples.epochMillis().length];
+ for (int round = 0; round < seconds.length; round++) {
+ seconds[round] = (samples.epochMillis()[round] - start) / 1000.0;
+ }
+ String name = label != null ? label : run.getParent().getFileName().toString();
+ return new RunData(name, name + " " + run.getFileName(), finished, consumersFinished(run, start), seconds,
+ samples.published(), sum(samples.dispatched(), seconds.length), sum(samples.backlog(), seconds.length),
+ latencies(run));
+ }
+
+ // When the last application received its last measured message, from the applications' summaries
+ private static double consumersFinished(Path run, long start) throws IOException {
+ JsonNode workload = workload(run);
+ ObjectMapper mapper = new ObjectMapper();
+ long last = -1;
+ for (int application = 0; ; application++) {
+ Path summary = RunReport.applicationDirectory(run, workload, application)
+ .resolve("application-summary.json");
+ if (!Files.isRegularFile(summary)) {
+ break;
+ }
+ last = Math.max(last, mapper.readTree(summary.toFile()).path("lastMeasurementMessageReceivedEpochMs")
+ .asLong(-1));
+ }
+ return last < 0 ? Double.NaN : (last - start) / 1000.0;
+ }
+
+ // The run's IoT workload settings, which name the applications' directories
+ private static JsonNode workload(Path run) throws IOException {
+ Path resolvedConfig = run.resolve(RunReport.RESOLVED_CONFIG);
+ return Files.isRegularFile(resolvedConfig)
+ ? new YAMLMapper().readTree(resolvedConfig.toFile()).path("workloads").path("iotTelemetry")
+ : MissingNode.getInstance();
+ }
+
+ /**
+ * Writes the charts.
+ *
+ * @param labelsInFileNames whether the file names include {@code -<baselineLabel>-vs-<comparisonLabel>}
+ * @return the charts written: each chart's combined form followed by its separate form
+ */
+ static List<Path> render(RunData a, RunData b, Path output, boolean labelsInFileNames) throws IOException {
+ Files.createDirectories(output);
+ String labels = labelsInFileNames ? "-" + fileNamePart(a.label()) + "-vs-" + fileNamePart(b.label()) : "";
+ String footer = "A: " + a.footer() + " B: " + b.footer();
+ double end = Math.max(last(a.seconds()), last(b.seconds()));
+ XAxis time = XAxis.seconds(0, end);
+ List<Line> throughput = List.of(
+ new Line("Published", Side.A, a.seconds(), a.published(), true),
+ new Line("Consumed", Side.A, a.seconds(), a.consumed(), false),
+ new Line("Published", Side.B, b.seconds(), b.published(), true),
+ new Line("Consumed", Side.B, b.seconds(), b.consumed(), false));
+ List<Line> backlog = List.of(
+ new Line("Backlog", Side.A, a.seconds(), a.backlog(), false),
+ new Line("Backlog", Side.B, b.seconds(), b.backlog(), false));
+ List<Line> latency = new ArrayList<>();
+ addCurves(latency, a, Side.A);
+ addCurves(latency, b, Side.B);
+ XAxis percentiles = XAxis.percentiles(HdrHistogramRenderer.percentileAxisPosition(MAX_PERCENTILE));
+ double latencyMax = TimeSeriesRenderer.axisMaximum(max(latency, percentiles) * 1.05);
+ List<Marker> markers = List.of(
+ new Marker(Side.A, Event.GATEWAYS_FINISHED, a.gatewaysFinishedSeconds()),
+ new Marker(Side.A, Event.CONSUMERS_FINISHED, a.consumersFinishedSeconds()),
+ new Marker(Side.B, Event.GATEWAYS_FINISHED, b.gatewaysFinishedSeconds()),
+ new Marker(Side.B, Event.CONSUMERS_FINISHED, b.consumersFinishedSeconds()));
+ List<Path> written = new ArrayList<>();
+ write(written, output, THROUGHPUT + labels, new Chart("Throughput",
+ "Messages per second, sampled once per second", time,
+ TimeSeriesRenderer.axisMaximum(max(throughput, time) * 1.05), false, 0, throughput, markers),
+ a, b, footer);
+ write(written, output, BACKLOG + labels, new Chart("Backlog",
+ "Messages in the backlog, sampled once per second", time,
+ TimeSeriesRenderer.axisMaximum(max(backlog, time) * 1.05), false, 0, backlog, markers),
+ a, b, footer);
+ write(written, output, LATENCY + labels, new Chart("Latency by percentile", "Latency (ms)", percentiles,
+ latencyMax, false, 0, latency, List.of()), a, b, footer);
+ write(written, output, LATENCY_LOG + labels, new Chart("Latency by percentile",
+ "Latency (ms), logarithmic scale", percentiles, Math.pow(10, Math.ceil(Math.log10(latencyMax))), true,
+ LOG_LATENCY_MIN, latency, List.of()), a, b, footer);
+ return written;
+ }
+
+ // A label as part of a file name: letters, digits, dots and underscores, anything else as an underscore
+ static String fileNamePart(String label) {
+ return label.replaceAll("[^A-Za-z0-9._]", "_");
+ }
+
+ private static void write(List<Path> written, Path output, String name, Chart chart, RunData a, RunData b,
+ String footer) throws IOException {
+ Path combined = output.resolve(name + ".svg");
+ Path separate = output.resolve(name + SEPARATE_SUFFIX + ".svg");
+ Files.writeString(combined, ComparisonRenderer.combined(chart, a.label(), b.label(), footer));
+ Files.writeString(separate, ComparisonRenderer.separate(chart, a.label(), b.label(), footer));
+ written.add(combined);
+ written.add(separate);
+ }
+
+ private static void addCurves(List<Line> lines, RunData run, Side side) {
+ for (int index = 0; index < run.latencies().size(); index++) {
+ Curve curve = run.latencies().get(index);
+ // Publish dashed, as the published rate is in the throughput chart
+ lines.add(new Line(curve.name(), side, curve.positions(), curve.millis(), index == 0));
+ }
+ }
+
+ // The publish latency, then each application's end-to-end latency; one application is "End-to-end"
+ private static List<Curve> latencies(Path run) throws IOException {
+ List<Curve> curves = new ArrayList<>();
+ Path publish = run.resolve("gateways/gateways-latency.hdr");
+ if (!Files.isRegularFile(publish)) {
+ return curves;
+ }
+ curves.add(curve("Publish", publish));
+ JsonNode workload = workload(run);
+ List<Path> logs = new ArrayList<>();
+ List<String> names = new ArrayList<>();
+ for (int application = 0; ; application++) {
+ Path directory = RunReport.applicationDirectory(run, workload, application);
+ Path log = directory.resolve("application-latency.hdr");
+ if (!Files.isRegularFile(log)) {
+ break;
+ }
+ logs.add(log);
+ names.add(directory.getFileName().toString());
+ }
+ for (int index = 0; index < logs.size(); index++) {
+ curves.add(curve(logs.size() == 1 ? "End-to-end" : "End-to-end, " + names.get(index), logs.get(index)));
+ }
+ return curves;
+ }
+
+ private static Curve curve(String name, Path log) throws IOException {
+ Histogram histogram = HdrHistogramRenderer.readMerged(List.of(log));
+ List<Double> positions = new ArrayList<>();
+ List<Double> millis = new ArrayList<>();
+ for (HistogramIterationValue value : histogram.percentiles(PERCENTILE_TICKS_PER_HALF_DISTANCE)) {
+ double percentile = Math.min(value.getPercentileLevelIteratedTo(), MAX_PERCENTILE);
+ positions.add(HdrHistogramRenderer.percentileAxisPosition(percentile));
+ millis.add(value.getValueIteratedTo() / 1000.0);
+ if (percentile >= MAX_PERCENTILE) {
+ break;
+ }
+ }
+ return new Curve(name, positions.stream().mapToDouble(Double::doubleValue).toArray(),
+ millis.stream().mapToDouble(Double::doubleValue).toArray());
+ }
+
+ // The subscriptions' values summed per sample, NaN where none has a value
+ private static double[] sum(Map<String, double[]> bySubscription, int rounds) {
+ double[] total = new double[rounds];
+ Arrays.fill(total, Double.NaN);
+ for (double[] values : bySubscription.values()) {
+ for (int round = 0; round < rounds; round++) {
+ if (!Double.isNaN(values[round])) {
+ total[round] = (Double.isNaN(total[round]) ? 0 : total[round]) + values[round];
+ }
+ }
+ }
+ return total;
+ }
+
+ // The largest value of the lines within the x axis, at least 1
+ private static double max(List<Line> lines, XAxis axis) {
+ double max = 1;
+ for (Line line : lines) {
+ for (int index = 0; index < line.x().length; index++) {
+ if (line.x()[index] >= axis.min() && line.x()[index] <= axis.max() && !Double.isNaN(line.y()[index])) {
+ max = Math.max(max, line.y()[index]);
+ }
+ }
+ }
+ return max;
+ }
+
+ private static double last(double[] values) {
+ return values.length == 0 ? 0 : values[values.length - 1];
+ }
+}
diff --git a/tests/performance/report-tool/src/main/java/org/apache/pulsar/tests/performance/report/ComparisonRenderer.java b/tests/performance/report-tool/src/main/java/org/apache/pulsar/tests/performance/report/ComparisonRenderer.java
new file mode 100644
index 000000000000..c9ad5b4a9784
--- /dev/null
+++ b/tests/performance/report-tool/src/main/java/org/apache/pulsar/tests/performance/report/ComparisonRenderer.java
@@ -0,0 +1,414 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.tests.performance.report;
+
+import static org.apache.pulsar.tests.performance.report.ChartStyle.GRID;
+import static org.apache.pulsar.tests.performance.report.ChartStyle.INK;
+import static org.apache.pulsar.tests.performance.report.ChartStyle.MUTED;
+import static org.apache.pulsar.tests.performance.report.ChartStyle.WIDTH;
+import static org.apache.pulsar.tests.performance.report.ChartStyle.xml;
+import java.awt.Color;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Locale;
+
+/**
+ * Draws a chart that compares a baseline run (A) with a comparison run (B), as SVG, in the width and style of the run
+ * report's charts. The same chart is drawn in two forms:
+ * <ul>
+ * <li>combined: both runs' lines in one diagram, A in {@link #A_COLOR} and B in {@link #B_COLOR};</li>
+ * <li>separate: A above B, in two panels with the same axes, each tinted in its run's color and labeled with it,
+ * so that the runs can be told apart at a glance and their values compared by position.</li>
+ * </ul>
+ * The lines are thin in both forms: the colors tell the runs apart.
+ * Each run can have several lines, such as the published and consumed rates; within a run they differ by dash.
+ */
+final class ComparisonRenderer {
+ /** The baseline run's color: blue. */
+ static final Color A_COLOR = new Color(52, 101, 164);
+ /** The comparison run's color: orange, which stays distinct from the blue for color-blind readers. */
+ static final Color B_COLOR = new Color(214, 95, 0);
+ /** The separate charts' panel backgrounds: light tints of the runs' colors. */
+ static final Color A_TINT = new Color(236, 242, 250);
+ static final Color B_TINT = new Color(253, 241, 230);
+ /** Every line's width, for both runs. */
+ static final double STROKE = 1.5;
+ private static final int PLOT_LEFT = 110;
+ private static final int PLOT_RIGHT = WIDTH - 50;
+ private static final int TITLE_HEIGHT = 110;
+ private static final int COMBINED_PLOT_HEIGHT = 340;
+ private static final int PANEL_PLOT_HEIGHT = 230;
+ // Below each plot: the tick labels and the x axis title
+ private static final int X_AXIS_HEIGHT = 56;
+ // Above each panel of a separate chart: its run's badge
+ private static final int BADGE_HEIGHT = 26;
+ private static final int PANEL_GAP = BADGE_HEIGHT + 20;
+ private static final int LEGEND_ROW_HEIGHT = 24;
+ private static final int FOOTER_HEIGHT = 30;
+ private static final int GRID_LINES = 5;
+ private static final String PUBLISHED_DASHES = "9 6";
+ // A marker label's width per character at its font size, and the distance between stacked labels
+ private static final double MARKER_CHAR_WIDTH = 7.2;
+ private static final int MARKER_ROW_HEIGHT = 16;
+
+ /** The run a line belongs to. */
+ enum Side {
+ A, B
+ }
+
+ /**
+ * One line; a {@link Double#NaN} value leaves a gap.
+ *
+ * @param dashed whether the line is dashed, which tells a run's lines apart, such as published from consumed
+ */
+ record Line(String name, Side side, double[] x, double[] y, boolean dashed) {
+ }
+
+ /**
+ * The x axis: seconds since the measurement start, or percentiles spread as
+ * {@link HdrHistogramRenderer#percentileAxisPosition(double)} does, on a logarithmic scale.
+ */
+ record XAxis(String title, boolean percentile, double min, double max) {
+ static XAxis seconds(double min, double max) {
+ return new XAxis("Seconds since the measurement start", false, min, max);
+ }
+
+ static XAxis percentiles(double maxPosition) {
+ return new XAxis("Percentile", true, 1, maxPosition);
+ }
+ }
+
+ /** What a run's marker line shows: when its gateways or its consumers finished. */
+ enum Event {
+ GATEWAYS_FINISHED("gateways finished", "5 4"),
+ CONSUMERS_FINISHED("consumers finished", "1.5 3");
+
+ private final String label;
+ private final String dashes;
+
+ Event(String label, String dashes) {
+ this.label = label;
+ this.dashes = dashes;
+ }
+ }
+
+ /**
+ * A vertical marker line of a run, labeled with its event.
+ *
+ * @param at the x position, {@link Double#NaN} for none
+ */
+ record Marker(Side side, Event event, double at) {
+ }
+
+ /**
+ * What a chart shows, for both runs.
+ *
+ * @param yMax the top of the value axis, the same for both runs
+ * @param logY whether the value axis is logarithmic, from {@code yMinLog} to {@code yMax}
+ */
+ record Chart(String title, String yLabel, XAxis x, double yMax, boolean logY, double yMinLog, List<Line> lines,
+ List<Marker> markers) {
+ }
+
+ private record Panel(int top, int height) {
+ int bottom() {
+ return top + height;
+ }
+ }
+
+ private ComparisonRenderer() {
+ }
+
+ /** The combined form: both runs' lines in one diagram. */
+ static String combined(Chart chart, String labelA, String labelB, String footer) {
+ Panel panel = new Panel(TITLE_HEIGHT, COMBINED_PLOT_HEIGHT);
+ List<String> legend = new ArrayList<>();
+ int legendTop = panel.bottom() + X_AXIS_HEIGHT + 10;
+ int legendRows = legend(legend, chart.lines(), labelA, labelB, true, legendTop);
+ int height = legendTop + legendRows * LEGEND_ROW_HEIGHT + FOOTER_HEIGHT;
+ StringBuilder out = start(chart, height, "A: " + labelA + " · B: " + labelB);
+ plot(out, chart, panel, null, chart.lines());
+ markers(out, chart, panel, chart.markers(), true);
+ legend.forEach(out::append);
+ return end(out, footer, height);
+ }
+
+ /** The separate form: A above B, with the same axes, each in a panel tinted in its run's color. */
+ static String separate(Chart chart, String labelA, String labelB, String footer) {
+ Panel panelA = new Panel(TITLE_HEIGHT + BADGE_HEIGHT + 8, PANEL_PLOT_HEIGHT);
+ Panel panelB = new Panel(panelA.bottom() + X_AXIS_HEIGHT + PANEL_GAP, PANEL_PLOT_HEIGHT);
+ List<String> legend = new ArrayList<>();
+ int legendTop = panelB.bottom() + X_AXIS_HEIGHT + 10;
+ int legendRows = legend(legend, chart.lines(), labelA, labelB, false, legendTop);
+ int height = legendTop + legendRows * LEGEND_ROW_HEIGHT + FOOTER_HEIGHT;
+ StringBuilder out = start(chart, height, "A: " + labelA + " · B: " + labelB + " · both panels have the same"
+ + " axes");
+ plot(out, chart, panelA, Side.A, chart.lines().stream().filter(line -> line.side() == Side.A).toList());
+ badge(out, panelA, Side.A, labelA);
+ markers(out, chart, panelA, chart.markers().stream().filter(marker -> marker.side() == Side.A).toList(),
+ false);
+ plot(out, chart, panelB, Side.B, chart.lines().stream().filter(line -> line.side() == Side.B).toList());
+ badge(out, panelB, Side.B, labelB);
+ markers(out, chart, panelB, chart.markers().stream().filter(marker -> marker.side() == Side.B).toList(),
+ false);
+ legend.forEach(out::append);
+ return end(out, footer, height);
+ }
+
+ private static StringBuilder start(Chart chart, int height, String subtitle) {
+ StringBuilder out = new StringBuilder(32_000);
+ out.append("<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 ").append(WIDTH).append(' ').append(height)
+ .append("\">\n<rect width=\"100%\" height=\"100%\" fill=\"white\"/>\n")
+ .append("<style>text{font-family:DejaVu Sans,Arial,sans-serif;fill:").append(hex(INK))
+ .append("}.muted{fill:").append(hex(MUTED)).append("}</style>\n")
+ .append("<text x=\"70\" y=\"55\" font-size=\"28\" font-weight=\"bold\">").append(xml(chart.title()))
+ .append("</text>\n<text class=\"muted\" x=\"70\" y=\"82\" font-size=\"15\">")
+ .append(xml(chart.yLabel() + " · " + subtitle)).append("</text>\n");
+ return out;
+ }
+
+ private static String end(StringBuilder out, String footer, int height) {
+ ChartStyle.appendSvgFooter(out, footer, height);
+ return out.append("</svg>\n").toString();
+ }
+
+ // The grid, the axes and the lines of one plot; a panel of a separate chart gets its run's tint
+ private static void plot(StringBuilder out, Chart chart, Panel panel, Side tint, List<Line> lines) {
+ if (tint != null) {
+ rect(out, PLOT_LEFT, panel.top(), PLOT_RIGHT - PLOT_LEFT, panel.height(),
+ tint == Side.A ? A_TINT : B_TINT);
+ }
+ for (double value : yTicks(chart)) {
+ int y = y(chart, panel, value);
+ out.append("<line x1=\"").append(PLOT_LEFT).append("\" y1=\"").append(y).append("\" x2=\"")
+ .append(PLOT_RIGHT).append("\" y2=\"").append(y).append("\" stroke=\"").append(hex(GRID))
+ .append("\"/>\n<text class=\"muted\" x=\"").append(PLOT_LEFT - 10).append("\" y=\"")
+ .append(y + 4).append("\" text-anchor=\"end\" font-size=\"12\">")
+ .append(TimeSeriesRenderer.compact(value)).append("</text>\n");
+ }
+ for (double tick : xTicks(chart.x())) {
+ int x = x(chart.x(), tick);
+ out.append("<line x1=\"").append(x).append("\" y1=\"").append(panel.bottom()).append("\" x2=\"")
+ .append(x).append("\" y2=\"").append(panel.bottom() + 5).append("\" stroke=\"")
+ .append(hex(INK)).append("\"/>\n<text x=\"").append(x).append("\" y=\"")
+ .append(panel.bottom() + 20).append("\" text-anchor=\"middle\" font-size=\"12\">")
+ .append(xml(xLabel(chart.x(), tick))).append("</text>\n");
+ }
+ out.append("<line x1=\"").append(PLOT_LEFT).append("\" y1=\"").append(panel.bottom()).append("\" x2=\"")
+ .append(PLOT_RIGHT).append("\" y2=\"").append(panel.bottom()).append("\" stroke=\"").append(hex(INK))
+ .append("\"/>\n<text x=\"").append((PLOT_LEFT + PLOT_RIGHT) / 2).append("\" y=\"")
+ .append(panel.bottom() + 42).append("\" text-anchor=\"middle\" font-size=\"12\">")
+ .append(xml(chart.x().title())).append("</text>\n");
+ // A first, so that B's lines are drawn on top
+ for (Side side : Side.values()) {
+ for (Line line : lines) {
+ if (line.side() == side) {
+ polylines(out, chart, panel, line);
+ }
+ }
+ }
+ }
+
+ private static void polylines(StringBuilder out, Chart chart, Panel panel, Line line) {
+ List<StringBuilder> segments = new ArrayList<>();
+ StringBuilder current = null;
+ for (int index = 0; index < line.x().length; index++) {
+ double xValue = line.x()[index];
+ double yValue = line.y()[index];
+ boolean defined = !Double.isNaN(xValue) && !Double.isNaN(yValue) && xValue >= chart.x().min()
+ && xValue <= chart.x().max() && (!chart.logY() || yValue > 0);
+ if (!defined) {
+ current = null;
+ continue;
+ }
+ if (current == null) {
+ current = new StringBuilder();
+ segments.add(current);
+ }
+ current.append(x(chart.x(), xValue)).append(',').append(y(chart, panel, yValue)).append(' ');
+ }
+ for (StringBuilder segment : segments) {
+ out.append("<polyline fill=\"none\" stroke=\"").append(hex(color(line.side())))
+ .append("\" stroke-width=\"").append(STROKE).append("\" stroke-linejoin=\"round\"")
+ .append(line.dashed() ? " stroke-dasharray=\"" + PUBLISHED_DASHES + "\"" : "")
+ .append(" points=\"").append(segment).append("\"/>\n");
+ }
+ }
+
+ // The run's name in its color above its panel, where it doesn't cover the lines, so that a panel says which run
+ // it shows
+ private static void badge(StringBuilder out, Panel panel, Side side, String label) {
+ String text = side + ": " + label;
+ int width = 24 + (int) (text.length() * 8.2);
+ int top = panel.top() - BADGE_HEIGHT - 4;
+ rect(out, PLOT_LEFT, top, width, BADGE_HEIGHT, color(side));
+ out.append("<text x=\"").append(PLOT_LEFT + 12).append("\" y=\"").append(top + 18)
+ .append("\" font-size=\"14\" font-weight=\"bold\" style=\"fill:white\">").append(xml(text))
+ .append("</text>\n");
+ }
+
+ /**
+ * Draws the markers' lines and labels. A label goes right of its line, or left of it near the plot's right edge,
+ * in the first row from the top where it doesn't overlap another marker's label, so that labels of close markers
+ * stack instead of overlapping.
+ *
+ * @param prefix whether a label names its run, as in a combined chart
+ */
+ private static void markers(StringBuilder out, Chart chart, Panel panel, List<Marker> markers, boolean prefix) {
+ List<Marker> shown = markers.stream().filter(marker -> !Double.isNaN(marker.at())
+ && marker.at() >= chart.x().min() && marker.at() <= chart.x().max())
+ .sorted(Comparator.comparingDouble(Marker::at)).toList();
+ List<List<int[]>> rows = new ArrayList<>();
+ for (Marker marker : shown) {
+ int x = x(chart.x(), marker.at());
+ String label = (prefix ? marker.side() + ": " : "") + marker.event().label;
+ int width = (int) Math.ceil(label.length() * MARKER_CHAR_WIDTH);
+ boolean left = x + 6 + width > PLOT_RIGHT;
+ int[] extent = left ? new int[] {x - 6 - width, x} : new int[] {x, x + 6 + width};
+ int row = 0;
+ while (row < rows.size() && overlaps(rows.get(row), extent)) {
+ row++;
+ }
+ if (row == rows.size()) {
+ rows.add(new ArrayList<>());
+ }
+ rows.get(row).add(extent);
+ String color = hex(color(marker.side()));
+ out.append("<line x1=\"").append(x).append("\" y1=\"").append(panel.top()).append("\" x2=\"").append(x)
+ .append("\" y2=\"").append(panel.bottom()).append("\" stroke=\"").append(color)
+ .append("\" stroke-width=\"1.5\" stroke-dasharray=\"").append(marker.event().dashes)
+ .append("\"/>\n<text x=\"").append(left ? x - 6 : x + 6).append("\" y=\"")
+ .append(panel.top() + 14 + row * MARKER_ROW_HEIGHT).append(left ? "\" text-anchor=\"end" : "")
+ .append("\" font-size=\"12\" style=\"fill:").append(color).append("\">").append(xml(label))
+ .append("</text>\n");
+ }
+ }
+
+ // Whether an extent, with a small gap, overlaps any of a row's extents
+ private static boolean overlaps(List<int[]> row, int[] extent) {
+ for (int[] other : row) {
+ if (extent[0] < other[1] + 8 && other[0] < extent[1] + 8) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Appends the legend: a combined chart lists each run's lines with their run, a separate chart each run's lines
+ * in the run's color.
+ *
+ * @return the number of rows
+ */
+ private static int legend(List<String> out, List<Line> lines, String labelA, String labelB, boolean combined,
+ int top) {
+ int x = PLOT_LEFT;
+ int row = 0;
+ for (Side side : Side.values()) {
+ for (Line line : lines) {
+ if (line.side() != side) {
+ continue;
+ }
+ String text = (combined ? side + " (" + (side == Side.A ? labelA : labelB) + "): " : side + ": ")
+ + line.name();
+ int width = 56 + (int) (text.length() * 7.8);
+ if (x > PLOT_LEFT && x + width > PLOT_RIGHT) {
+ x = PLOT_LEFT;
+ row++;
+ }
+ int y = top + row * LEGEND_ROW_HEIGHT + 14;
+ out.add("<line x1=\"" + x + "\" y1=\"" + (y - 5) + "\" x2=\"" + (x + 28) + "\" y2=\"" + (y - 5)
+ + "\" stroke=\"" + hex(color(side)) + "\" stroke-width=\"" + STROKE + "\""
+ + (line.dashed() ? " stroke-dasharray=\"" + PUBLISHED_DASHES + "\"" : "") + "/>\n<text x=\""
+ + (x + 36) + "\" y=\"" + y + "\" font-size=\"14\">" + xml(text) + "</text>\n");
+ x += width;
+ }
+ }
+ return row + 1;
+ }
+
+ static Color color(Side side) {
+ return side == Side.A ? A_COLOR : B_COLOR;
+ }
+
+ static int x(XAxis axis, double value) {
+ double fraction = axis.percentile()
+ ? (Math.log10(value) - Math.log10(axis.min())) / (Math.log10(axis.max()) - Math.log10(axis.min()))
+ : (value - axis.min()) / (axis.max() - axis.min());
+ return PLOT_LEFT + (int) Math.round(fraction * (PLOT_RIGHT - PLOT_LEFT));
+ }
+
+ private static int y(Chart chart, Panel panel, double value) {
+ double fraction;
+ if (chart.logY()) {
+ double clamped = Math.max(value, chart.yMinLog());
+ fraction = (Math.log10(clamped) - Math.log10(chart.yMinLog()))
+ / (Math.log10(chart.yMax()) - Math.log10(chart.yMinLog()));
+ } else {
+ fraction = value / chart.yMax();
+ }
+ return panel.bottom() - (int) Math.round(fraction * panel.height());
+ }
+
+ // Linear: five steps from zero; logarithmic: one line per decade
+ static List<Double> yTicks(Chart chart) {
+ List<Double> ticks = new ArrayList<>();
+ if (chart.logY()) {
+ for (double value = chart.yMinLog(); value <= chart.yMax() * 1.0001; value *= 10) {
+ ticks.add(value);
+ }
+ } else {
+ for (int line = 0; line <= GRID_LINES; line++) {
+ ticks.add(chart.yMax() * line / GRID_LINES);
+ }
+ }
+ return ticks;
+ }
+
+ // Seconds: a round step for about ten ticks; percentiles: each nine
+ static List<Double> xTicks(XAxis axis) {
+ List<Double> ticks = new ArrayList<>();
+ if (axis.percentile()) {
+ for (double position = 1; position <= axis.max() * 1.0001; position *= 10) {
+ ticks.add(position);
+ }
+ } else {
+ double step = TimeSeriesRenderer.niceCeiling((axis.max() - axis.min()) / 10);
+ for (double tick = Math.ceil(axis.min() / step) * step; tick <= axis.max() + 1e-9; tick += step) {
+ ticks.add(tick);
+ }
+ }
+ return ticks;
+ }
+
+ private static String xLabel(XAxis axis, double tick) {
+ return axis.percentile() ? HdrHistogramRenderer.percentileAxisLabel(tick)
+ : String.format(Locale.ROOT, "%.0f", tick);
+ }
+
+ private static void rect(StringBuilder out, int x, int y, int width, int height, Color fill) {
+ out.append("<rect x=\"").append(x).append("\" y=\"").append(y).append("\" width=\"").append(width)
+ .append("\" height=\"").append(height).append("\" fill=\"").append(hex(fill)).append("\"/>\n");
+ }
+
+ static String hex(Color color) {
+ return String.format(Locale.ROOT, "#%02x%02x%02x", color.getRed(), color.getGreen(), color.getBlue());
+ }
+}
diff --git a/tests/performance/report-tool/src/test/java/org/apache/pulsar/tests/performance/report/ComparisonChartsTest.java b/tests/performance/report-tool/src/test/java/org/apache/pulsar/tests/performance/report/ComparisonChartsTest.java
new file mode 100644
index 000000000000..47677c089df2
--- /dev/null
+++ b/tests/performance/report-tool/src/test/java/org/apache/pulsar/tests/performance/report/ComparisonChartsTest.java
@@ -0,0 +1,206 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.tests.performance.report;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import com.fasterxml.jackson.databind.node.MissingNode;
+import java.io.IOException;
+import java.io.PrintStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Stream;
+import org.HdrHistogram.Histogram;
+import org.HdrHistogram.HistogramLogWriter;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+public class ComparisonChartsTest {
+ private static final long START = 1_000_000;
+ private Path directory;
+
+ @BeforeMethod
+ public void createDirectory() throws IOException {
+ directory = Files.createTempDirectory("comparison-charts-test");
+ }
+
+ @AfterMethod(alwaysRun = true)
+ public void deleteDirectory() throws IOException {
+ try (Stream<Path> paths = Files.walk(directory)) {
+ for (Path path : paths.sorted(Comparator.reverseOrder()).toList()) {
+ Files.delete(path);
+ }
+ }
+ }
+
+ @Test
+ public void writesCombinedAndSeparateChartsWithTheSameAxesForBothRuns() throws Exception {
+ // A publishes 1,000 msg/s for 10 s and its consumers receive nothing; B publishes and consumes 4,000 msg/s
+ // for 5 s
+ Path a = writeRun("baseline/run-a", 1_000, 0, 10, 2_000, 200_000);
+ Path b = writeRun("change/run-b", 4_000, 4_000, 5, 1_000, 2_000);
+
+ List<Path> charts = ComparisonCharts.render(ComparisonCharts.read(a, "4.0.13"),
+ ComparisonCharts.read(b, null), directory.resolve("charts"), false);
+
+ List<String> names = charts.stream().map(path -> path.getFileName().toString()).toList();
+ assertThat(names).containsExactly("throughput.svg", "throughput-separate.svg", "backlog.svg",
+ "backlog-separate.svg", "latency-percentiles.svg", "latency-percentiles-separate.svg",
+ "latency-percentiles-log.svg", "latency-percentiles-log-separate.svg");
+ String combined = Files.readString(directory.resolve("charts/throughput.svg"));
+ // The runs' labels: given for A, the run's name, its directory's parent, for B
+ assertThat(combined).contains("A (4.0.13): Published").contains("B (change): Consumed");
+ // A's lines are blue and B's orange, both thin
+ String aColor = ComparisonRenderer.hex(ComparisonRenderer.A_COLOR);
+ String bColor = ComparisonRenderer.hex(ComparisonRenderer.B_COLOR);
+ assertThat(combined).contains("stroke=\"" + aColor + "\" stroke-width=\"" + ComparisonRenderer.STROKE)
+ .contains("stroke=\"" + bColor + "\" stroke-width=\"" + ComparisonRenderer.STROKE);
+ // The value axis reaches past B's 4,000 msg/s in both forms, and the separate form's panels share it
+ String separate = Files.readString(directory.resolve("charts/throughput-separate.svg"));
+ assertThat(yAxisLabels(combined)).containsExactly("0", "1k", "2k", "3k", "4k", "5k");
+ assertThat(yAxisLabels(separate)).containsExactly("0", "1k", "2k", "3k", "4k", "5k", "0", "1k", "2k", "3k",
+ "4k", "5k");
+ // Each run's gateways and consumers finished, A's consumers 12 s after the start
+ assertThat(combined).contains(">A: gateways finished</text>").contains(">A: consumers finished</text>")
+ .contains(">B: gateways finished</text>").contains(">B: consumers finished</text>");
+ assertThat(separate).contains(">gateways finished</text>").contains(">consumers finished</text>");
+ // Each panel is tinted in its run's color and labeled with it
+ assertThat(separate).contains("fill=\"" + ComparisonRenderer.hex(ComparisonRenderer.A_TINT) + "\"")
+ .contains("fill=\"" + ComparisonRenderer.hex(ComparisonRenderer.B_TINT) + "\"")
+ .contains(">A: 4.0.13</text>").contains(">B: change</text>");
+ // The logarithmic latency axis spans A's 200 ms and B's 2 ms by decades
+ String log = Files.readString(directory.resolve("charts/latency-percentiles-log.svg"));
+ assertThat(yAxisLabels(log)).containsExactly("0.1", "1", "10", "100", "1k");
+ assertThat(Files.readString(directory.resolve("charts/latency-percentiles.svg")))
+ .contains("A (4.0.13): End-to-end").contains("B (change): Publish");
+ }
+
+ @Test
+ public void namesTheChartsAfterTheRunsLabels() throws Exception {
+ Path a = writeRun("baseline/run-a", 1_000, 0, 10, 2_000, 200_000);
+ Path b = writeRun("change/run-b", 4_000, 4_000, 5, 1_000, 2_000);
+
+ List<Path> charts = ComparisonCharts.render(ComparisonCharts.read(a, "4.0.13"),
+ ComparisonCharts.read(b, "lh/branch x"), directory.resolve("charts"), true);
+
+ assertThat(charts.stream().map(path -> path.getFileName().toString()).toList().subList(0, 2))
+ .containsExactly("throughput-4.0.13-vs-lh_branch_x.svg",
+ "throughput-4.0.13-vs-lh_branch_x-separate.svg");
+ }
+
+ @Test
+ public void stacksTheLabelsOfCloseMarkersInRowsOfTheirOwn() {
+ ComparisonRenderer.Chart chart = new ComparisonRenderer.Chart("Throughput", "Messages per second",
+ ComparisonRenderer.XAxis.seconds(0, 100), 10, false, 0, List.of(), List.of(
+ new ComparisonRenderer.Marker(ComparisonRenderer.Side.A,
+ ComparisonRenderer.Event.GATEWAYS_FINISHED, 30),
+ new ComparisonRenderer.Marker(ComparisonRenderer.Side.A,
+ ComparisonRenderer.Event.CONSUMERS_FINISHED, 31),
+ new ComparisonRenderer.Marker(ComparisonRenderer.Side.B,
+ ComparisonRenderer.Event.GATEWAYS_FINISHED, 32),
+ new ComparisonRenderer.Marker(ComparisonRenderer.Side.B,
+ ComparisonRenderer.Event.CONSUMERS_FINISHED, 90)));
+
+ String svg = ComparisonRenderer.combined(chart, "4.0.13", "5.0.0", "");
+
+ // The three close labels are in three rows; B's consumers' label, far from them, goes back to the first
+ Matcher matcher = Pattern.compile("y=\"(\\d+)\"(?: text-anchor=\"end\")? font-size=\"12\""
+ + " style=\"fill:#[0-9a-f]{6}\">([AB]: [a-z ]+)</text>").matcher(svg);
+ Map<String, Integer> rows = new HashMap<>();
+ while (matcher.find()) {
+ rows.put(matcher.group(2), Integer.parseInt(matcher.group(1)));
+ }
+ assertThat(rows).hasSize(4);
+ assertThat(Set.of(rows.get("A: gateways finished"), rows.get("A: consumers finished"),
+ rows.get("B: gateways finished"))).hasSize(3);
+ assertThat(rows.get("B: consumers finished")).isEqualTo(rows.get("A: gateways finished"));
+ }
+
+ @Test
+ public void spreadsPercentilesByNinesAndSecondsByRoundSteps() {
+ ComparisonRenderer.XAxis percentiles = ComparisonRenderer.XAxis.percentiles(1_000_000);
+ assertThat(ComparisonRenderer.xTicks(percentiles))
+ .containsExactly(1.0, 10.0, 100.0, 1_000.0, 10_000.0, 100_000.0, 1_000_000.0);
+ assertThat(ComparisonRenderer.x(percentiles, 1_000)).isEqualTo((ComparisonRenderer.x(percentiles, 1)
+ + ComparisonRenderer.x(percentiles, 1_000_000)) / 2);
+ assertThat(ComparisonRenderer.xTicks(ComparisonRenderer.XAxis.seconds(0, 250)))
+ .containsExactly(0.0, 50.0, 100.0, 150.0, 200.0, 250.0);
+ }
+
+ // The value axis's labels, in the order they are drawn
+ private static List<String> yAxisLabels(String svg) {
+ Matcher matcher = Pattern.compile("text-anchor=\"end\" font-size=\"12\">([^<]+)</text>").matcher(svg);
+ List<String> labels = new ArrayList<>();
+ while (matcher.find()) {
+ labels.add(matcher.group(1));
+ }
+ return labels;
+ }
+
+ /**
+ * Writes a run directory: topic stats sampled once a second, the gateways' summary and latency logs, and one
+ * application's end-to-end latency log.
+ */
+ private Path writeRun(String name, int publishedPerSecond, int dispatchedPerSecond, int seconds,
+ long publishMicros, long endToEndMicros) throws Exception {
+ Path run = directory.resolve(name);
+ Files.createDirectories(run.resolve("gateways"));
+ StringBuilder csv = new StringBuilder("epochMillis,topic,subscription,msgBacklog,msgInCounter,msgOutCounter\n");
+ for (int second = 0; second <= seconds + 2; second++) {
+ long in = (long) publishedPerSecond * Math.min(second, seconds);
+ long out = (long) dispatchedPerSecond * Math.min(second, seconds);
+ csv.append(START + second * 1_000L).append(",persistent://public/default/t,sub,").append(in - out)
+ .append(',').append(in).append(',').append(out).append('\n');
+ }
+ Files.writeString(run.resolve(RunReport.TOPIC_STATS_FILE), csv);
+ Files.writeString(run.resolve("gateways/gateways-summary.json"), "{\"measurementStartEpochMs\":" + START
+ + ",\"measurementEndEpochMs\":" + (START + seconds * 1_000L) + "}");
+ Path application = RunReport.applicationDirectory(run, MissingNode.getInstance(), 0);
+ Files.createDirectories(application);
+ Files.writeString(application.resolve("application-summary.json"),
+ "{\"lastMeasurementMessageReceivedEpochMs\":" + (START + (seconds + 2) * 1_000L) + "}");
+ writeLog(run.resolve("gateways/gateways-latency.hdr"), publishMicros);
+ writeLog(RunReport.applicationDirectory(run, MissingNode.getInstance(), 0)
+ .resolve("application-latency.hdr"), endToEndMicros);
+ return run;
+ }
+
+ private static void writeLog(Path path, long micros) throws Exception {
+ Files.createDirectories(path.getParent());
+ Histogram histogram = new Histogram(3);
+ histogram.recordValue(micros / 2);
+ histogram.recordValue(micros);
+ histogram.setStartTimeStamp(START);
+ histogram.setEndTimeStamp(START + 1_000);
+ try (PrintStream output = new PrintStream(Files.newOutputStream(path))) {
+ HistogramLogWriter writer = new HistogramLogWriter(output);
+ writer.outputLogFormatVersion();
+ writer.outputLegend();
+ writer.outputIntervalHistogram(histogram);
+ }
+ }
+}
diff --git a/tests/performance/scenarios/README.md b/tests/performance/scenarios/README.md
index f3c121934d5e..35d0fbc32a35 100644
--- a/tests/performance/scenarios/README.md
+++ b/tests/performance/scenarios/README.md
@@ -107,6 +107,34 @@ dumps](../docs/heap-dumps.md) describes the other settings, such as dumps at giv
--args='--scenario tests/performance/scenarios/iot-telemetry.yaml --extends configs/heap-dumps-broker'
```
+### The bookies' journal on a tmpfs
+
+Every entry that a bookie stores is written to the disk twice: to its journal and to its entry log. All the bookies
+share the host's disk, so with large unbatched entries the disk's write bandwidth becomes the limit of a run on many
+developer machines, before Pulsar's: the max-rate scenario at 64 KB wrote 890–980 MB/s to a laptop's encrypted NVMe
+drive. `host-io.csv` and the run report's host section show when it is: the disk's write MB/s stays flat across runs
+while the host's CPU has headroom, as [the agent guide](../AGENTS.md#eliminating-bottlenecks) describes.
+
+[`bookie-journal-tmpfs.yaml`](configs/bookie-journal-tmpfs.yaml) puts each bookie's journal on a tmpfs of 2 GB with
+`cluster.bookies.journalTmpfs`, the tmpfs's size, so that only the ledger storage writes to the disk, which halves the
+disk's writes. The test cluster doesn't sync the journal anyway (`journalSyncData=false`), so it changes no
+durability guarantee of the test, only where the journal's writes go. In the max-rate scenario on that laptop it
+halved the disk's writes at 8 KB and 64 KB, and at 64 KB the broker's CPU per message fell 19 %.
+
+A bookie deletes its old journal files when it has flushed its write cache, but keeps the current file until it is
+full, so the launcher sets the journal's files to 256 MB without backups (`journalMaxSizeMB` and `journalMaxBackups`,
+in place of the scenario's), and each busy bookie's journal takes about 512 MB of memory. The bookies' `env` can't set
+`journalDirectory` with it. Compare runs only with the same setting.
+
+```bash
+./gradlew :tests:performance:launcher:run \
+ --args='--scenario tests/performance/scenarios/iot-telemetry-max-rate.yaml --extends configs/bookie-journal-tmpfs'
+```
+
+The ledger storage still writes every entry to the disk, and syncs its files each time it flushes its write cache.
+When those syncs are what's slow, [disabling write barriers](../environment/README.md#disabling-write-barriers) makes
+them cheaper on a disk without power loss protection, at the risk of the host's file system.
+
## Scenarios for the legacy TestNG runner
The scenarios of [the legacy TestNG profiling runner](../docs/legacy-testng-runner/README.md#scenario-files), which runs
diff --git a/tests/performance/scenarios/configs/bookie-journal-tmpfs.yaml b/tests/performance/scenarios/configs/bookie-journal-tmpfs.yaml
new file mode 100644
index 000000000000..4c1c0cd313e7
--- /dev/null
+++ b/tests/performance/scenarios/configs/bookie-journal-tmpfs.yaml
@@ -0,0 +1,25 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Puts each bookie's journal on a tmpfs of 2 GB, so that it isn't written to the disk. The test cluster doesn't sync
+# the journal, so this doesn't change what a crash would lose; it takes the journal's writes off the disk, which then
+# only has the ledger storage's. For runs whose limit is the disk's write bandwidth, such as the max-rate scenario with
+# large messages. The launcher sets the journal's files to 256 MB without backups, so that a bookie's journal takes
+# about 512 MB of memory. Add it with the launcher's --extends option.
+cluster:
+ bookies:
+ journalTmpfs: 2g
diff --git a/tests/performance/tools/src/main/dist/bin/run-workload b/tests/performance/tools/src/main/dist/bin/run-workload
index 1950a17645c6..865e39424f1c 100755
--- a/tests/performance/tools/src/main/dist/bin/run-workload
+++ b/tests/performance/tools/src/main/dist/bin/run-workload
@@ -64,6 +64,9 @@ if [[ $JAVA_MAJOR_VERSION -ge 24 ]]; then
# Netty, and the jonoffcpu agent in a profiled run, load native libraries, which is restricted from Java 24 on
OPTS="$OPTS --enable-native-access=ALL-UNNAMED"
fi
+# Netty's leak detection samples buffers and records stack traces, which costs CPU; bin/pulsar-perf turns it off by
+# default, and -Dio.netty.leakDetection.level in PULSAR_EXTRA_OPTS, which comes later on the command line, overrides it
+OPTS="-Dio.netty.leakDetection.level=disabled $OPTS"
OPTS="-XX:+ExitOnOutOfMemoryError -Dpulsar.allocator.exit_on_oom=true $OPTS"
OPTS="-Dio.netty.recycler.maxCapacityPerThread=4096 -Dio.netty.allocator.maxOrder=10 $OPTS"
OPTS="$OPTS $CALLER_PULSAR_MEM $CALLER_PULSAR_GC ${PULSAR_EXTRA_OPTS:-}"