You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@pulsar.apache.org by "lhotari (via GitHub)" <gi...@apache.org> on 2026/10/09 20:23:03 UTC

[PR] [improve][test] Compare two Pulsar images, or an image and the checkout, in the performance tests [pulsar]

lhotari opened a new pull request, #26847:
URL: https://github.com/apache/pulsar/pull/26847

### Motivation

The Pulsar Performance Testing Framework can already run a released Pulsar in the cluster with `-Pperformance.clusterPulsarImage`, and `compareRuns` can chart two runs against each other. Comparing two Pulsar versions, such as `apachepulsar/pulsar:4.0.14` and `apachepulsar/pulsar:5.0.0`, still takes several manual steps:

- running each side several times, alternating the order so that a drift of the host affects both sides alike;
- finding each side's median run;
- charting the median runs.

It also takes some care to keep the clients the same on both sides. This PR makes the whole comparison one Gradle command.

### Modifications

- Two new tasks in `:tests:performance:launcher`:
- `compareImages` compares two released Pulsar images, set with `-Pperformance.compare.baselineImage` and `-Pperformance.compare.candidateImage`.
- `compareImageWithCheckout` compares a released Pulsar image with this checkout.

Both take the launcher's arguments, such as `--scenario` and `--name`. Example:

```bash
./gradlew :tests:performance:launcher:compareImages \
-Pperformance.compare.baselineImage=apachepulsar/pulsar:4.0.14 \
-Pperformance.compare.candidateImage=apachepulsar/pulsar:5.0.0 \
--args='--scenario tests/performance/scenarios/iot-telemetry-high-rate.yaml --name release-ab'
```

- A new class, `ImageComparison`, runs the comparison:
- **Clusters:** each side's ZooKeeper, bookies and brokers run that side's Pulsar. A released image runs in the test image built on it, as with `-Pperformance.clusterPulsarImage`. The gateways and the applications always run the checkout's test image and its Pulsar client, so only the clusters differ.
- **Runs:** it runs the scenario `-Pperformance.compare.repetitions` times per side, 3 by default, alternating which side runs first (A B, B A, ...). Each run is a `PerformanceLauncher` in a JVM of its own, with a run directory of its own. If the comparison is interrupted, it stops the run in progress.
- **Median runs:** it leaves out failed runs, and runs whose applications received messages out of order or invalid messages. Of each side's other runs, it takes the median by `-Pperformance.compare.medianBy`: `throughput` (the default), `publish-p99`, `e2e-p99` or `broker-cpu`. For an even count, it takes the lower of the middle two.
- **Results:** it writes a `README.md` and an `index.html` to `<reports root>/<yyyy-MM-dd>/comparisons/<name>/<time>/`.
- The summary compares the median runs' throughput, publish p99, end-to-end p99 of the slowest application, and broker CPU per million messages, with the change from A to B. The p99 values are computed from the runs' HdrHistogram latency logs.
- It lists every run with its result and a link to its report, or to its console log when it failed.
- Beside it are the `compareRuns` charts of the two median runs, labeled with the images' tags or `checkout`.
- `:tests:java-test-image` builds the test images for both sides, named as `dockerBuildCluster` names its image.
- Small changes in the report tool:
- `ComparisonCharts.execute(String...)`, so that the comparison can call it without exiting the JVM;
- `MarkdownPages.renderHtml` is now public;
- `HdrHistogramRenderer.valueAtPercentileMillis` reads a percentile from latency logs.
- Docs:
- a new section, "Comparing two Pulsar images", in `tests/performance/docs/comparing-revisions.md`;
- the new Gradle properties and the comparisons directory in `running-scenarios.md`;
- short entries in `tests/performance/README.md` and `AGENTS.md`.

### Verifying this change

- [ ] Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

- `ImageComparisonTest` covers:
- the run order and the sides' labels;
- the launcher command for each side;
- reading a run's measures from its outputs, including the p99 from HdrHistogram logs and missing outputs;
- detecting runs with ordering violations or invalid messages;
- picking the median, leaving out failed and invalid runs and runs without the measure;
- the output directory, the summary's contents and the order of the charts.
- An end-to-end comparison of `apachepulsar/pulsar:4.0.14` and `apachepulsar/pulsar:5.0.0` on `iot-telemetry-small`, one run per side, wrote the summary with charts and working links to both runs' reports. A comparison of a released image with the checkout also worked.

### Does this pull request potentially affect one of the following parts:

*If the box was checked, please highlight the changes*

- [ ] Dependencies (add or upgrade a dependency)
- [ ] The public API
- [ ] The schema
- [ ] The default values of configurations
- [ ] The threading model
- [ ] The binary protocol
- [ ] The REST endpoints
- [ ] The admin CLI options
- [ ] The metrics
- [ ] Anything that affects deployment

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [improve][test] Compare two Pulsar images, or an image and the checkout, in the performance tests [pulsar]

Posted by "merlimat (via GitHub)" <gi...@apache.org>.

merlimat merged PR #26847:
URL: https://github.com/apache/pulsar/pull/26847


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org