You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by lh...@apache.org on 2026/10/12 19:18:27 UTC

(pulsar) 02/27: [improve][test] Compare two Pulsar images, or an image and the checkout, in the performance tests (#26847)

This is an automated email from the ASF dual-hosted git repository.

lhotari pushed a commit to branch branch-5.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit 1378def48a7eee80cfa56d92d12a9e600e36e175
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Fri Oct 9 19:23:27 2026 -0700

    [improve][test] Compare two Pulsar images, or an image and the checkout, in the performance tests (#26847)
    
    (cherry picked from commit f2f026c7db79317fb65dafb92b79b730dc83ada6)
---
 .../docker-images/java-test-image/build.gradle.kts |  14 +
 tests/performance/AGENTS.md                        |   2 +
 tests/performance/README.md                        |  13 +
 tests/performance/docs/comparing-revisions.md      |  41 ++
 tests/performance/docs/running-scenarios.md        |   5 +
 tests/performance/launcher/build.gradle.kts        |  59 ++-
 .../performance/launcher/ImageComparison.java      | 543 +++++++++++++++++++++
 .../performance/launcher/ImageComparisonTest.java  | 242 +++++++++
 .../tests/performance/report/ComparisonCharts.java |   7 +-
 .../performance/report/HdrHistogramRenderer.java   |  10 +
 .../tests/performance/report/MarkdownPages.java    |   2 +-
 11 files changed, 934 insertions(+), 4 deletions(-)

diff --git a/tests/docker-images/java-test-image/build.gradle.kts b/tests/docker-images/java-test-image/build.gradle.kts
index b98baae1b34c..76c76f1f0449 100644
--- a/tests/docker-images/java-test-image/build.gradle.kts
+++ b/tests/docker-images/java-test-image/build.gradle.kts
@@ -173,3 +173,17 @@ if (clusterPulsarImage != null) {
         description = "Build the java-test-image Docker image on the performance.clusterPulsarImage Pulsar image"
     }
 }
+
+// The clusters' images for :tests:performance:launcher:compareImages and compareImageWithCheckout, which compare two
+// released Pulsar images, -Pperformance.compare.baselineImage and -Pperformance.compare.candidateImage, or a released
+// image and this checkout: the test image built on each, named as dockerBuildCluster names its image.
+listOf("Baseline" to "performance.compare.baselineImage", "Candidate" to "performance.compare.candidateImage")
+    .forEach { (side, property) ->
+        providers.gradleProperty(property).orNull?.let { image ->
+            registerDockerBuild("dockerBuildCompare$side",
+                "cluster-" + image.replace(Regex("[^A-Za-z0-9_.-]"), "-").takeLast(120), "false", image, null)
+                .configure {
+                    description = "Build the java-test-image Docker image on the $property Pulsar image"
+                }
+        }
+    }
diff --git a/tests/performance/AGENTS.md b/tests/performance/AGENTS.md
index 6c3b6345dfb5..a4c9bb1c9b16 100644
--- a/tests/performance/AGENTS.md
+++ b/tests/performance/AGENTS.md
@@ -87,6 +87,8 @@ Run commands from the repository root. Gradle properties (`-P...`) configure the
 | Keep logs for diagnosis | Add `-Pperformance.keepLauncherLog` | Retains `launcher.log` even on success; failures retain it by default |
 | Start repetitions at a similar temperature | Add `-Pperformance.cooldownTemperature=<degrees-C>` | With Linux host sensors, waits before cluster startup and after warmup; keep the threshold identical across revisions and inspect timed-out waits in the report |
 | Disable metrics | Add `-Pperformance.metrics=false` | Omits metrics collection; keep this choice identical across compared runs |
+| Compare two released Pulsar images | `./gradlew :tests:performance:launcher:compareImages -Pperformance.compare.baselineImage=apachepulsar/pulsar:4.0.14 -Pperformance.compare.candidateImage=apachepulsar/pulsar:5.0.0 --args='--scenario <scenario> --name <experiment>'` | Alternates repeated runs of both clusters, picks each side's median run, and writes a summary and comparison charts, as [Comparing two Pulsar images](docs/comparing-revisions.md#comparing-two-pulsar-images) describes; th [...]
+| Compare a released image with the checkout | `./gradlew :tests:performance:launcher:compareImageWithCheckout -Pperformance.compare.baselineImage=apachepulsar/pulsar:5.0.0 --args='--scenario <scenario> --name <experiment>'` | As `compareImages`, with this checkout as the candidate |
 | Chart two runs against each other | `./gradlew :tests:performance:report-tool:compareRuns --args='--baseline <A run> --comparison <B run> --output <directory>'` | Writes throughput, backlog and latency charts with the same axes for both runs, combined and in separate panels, as [Comparison charts](docs/comparing-revisions.md#comparison-charts) describes |
 | Browse reports | `./gradlew :tests:performance:report-tool:serveReports` | Serves the configured reports root at <http://127.0.0.1:8000/> by default; stop the foreground server when finished |
 | Query stored metrics after a run | `./gradlew :tests:performance:metrics:up` | Starts VictoriaMetrics and Grafana in the background, prints URLs; default ports are 8428 and 3000 |
diff --git a/tests/performance/README.md b/tests/performance/README.md
index 0cd94e845fee..e9cf756f0812 100644
--- a/tests/performance/README.md
+++ b/tests/performance/README.md
@@ -431,6 +431,19 @@ such as the latest release or a particular one. The workloads, and so the Pulsar
 [Comparing with a released Pulsar](docs/comparing-revisions.md#comparing-with-a-released-pulsar) describes the
 details.
 
+To compare two released Pulsar images, or a released image and the checkout, in one command, use `compareImages` or
+`compareImageWithCheckout`. They alternate repeated runs of both clusters, pick each side's median run, and write a
+summary and comparison charts:
+
+```bash
+./gradlew :tests:performance:launcher:compareImages \
+  -Pperformance.compare.baselineImage=apachepulsar/pulsar:4.0.14 \
+  -Pperformance.compare.candidateImage=apachepulsar/pulsar:5.0.0 \
+  --args='--scenario tests/performance/scenarios/iot-telemetry-high-rate.yaml --name release-ab'
+```
+
+[Comparing two Pulsar images](docs/comparing-revisions.md#comparing-two-pulsar-images) describes them.
+
 When both revisions have valid runs, chart the median run of each against the other, with the same axes for both
 and each revision's label in the charts and their file names:
 
diff --git a/tests/performance/docs/comparing-revisions.md b/tests/performance/docs/comparing-revisions.md
index a693d5a13ac3..418772dc5bab 100644
--- a/tests/performance/docs/comparing-revisions.md
+++ b/tests/performance/docs/comparing-revisions.md
@@ -91,6 +91,47 @@ several times, so that a drift of the host, such as its temperature, affects bot
 Keep the scenario, its settings and the profiler options identical for both revisions. Profiling has a measurement
 cost, so compare profiled runs only with profiled runs.
 
+## Comparing two Pulsar images
+
+Two tasks run a whole comparison of two Pulsar clusters in one command:
+
+- **`compareImages`** compares two released Pulsar images, such as two releases.
+- **`compareImageWithCheckout`** compares a released Pulsar image with this checkout.
+
+```bash
+# Two releases
+./gradlew :tests:performance:launcher:compareImages \
+  -Pperformance.compare.baselineImage=apachepulsar/pulsar:4.0.14 \
+  -Pperformance.compare.candidateImage=apachepulsar/pulsar:5.0.0 \
+  --args='--scenario tests/performance/scenarios/iot-telemetry-high-rate.yaml --name release-ab'
+# A release and the checkout
+./gradlew :tests:performance:launcher:compareImageWithCheckout \
+  -Pperformance.compare.baselineImage=apachepulsar/pulsar:5.0.0 \
+  --args='--scenario tests/performance/scenarios/iot-telemetry-high-rate.yaml --name release-ab'
+```
+
+- **Clusters:** each side's ZooKeeper, bookies and brokers run its Pulsar: a released image in the test image built on
+  it, as [Comparing with a released Pulsar](#comparing-with-a-released-pulsar) describes, or the checkout's test image.
+  The gateways and the applications always run the checkout's test image, and with it its Pulsar client, so that only
+  the clusters differ.
+- **Runs:** the task runs the scenario `-Pperformance.compare.repetitions` times on each side, 3 by default, alternating
+  which side runs first (A B, B A, A B, ...), so that a drift of the host affects both sides alike. The arguments are
+  the launcher's; each run gets a run directory of its own, as with `run`, so `--output` isn't allowed.
+- **Median runs:** the task leaves out a run that fails, and a run whose applications received messages out of order or
+  invalid messages, since it measured a bug. Of each side's other runs, it takes the median by
+  `-Pperformance.compare.medianBy`: `throughput`, the default, `publish-p99`, `e2e-p99` or `broker-cpu`; for an even
+  count, the lower of the middle two. For a scenario with a rate limit, where the throughput is the same on both sides,
+  take the median by a latency or by the broker's CPU. The latencies are the 99th percentiles of the runs' latency
+  logs.
+- **Results:** the task writes the comparison to `<reports root>/<yyyy-MM-dd>/comparisons/<name>/<MM-dd-HH-mm-ss>/` and
+  prints its path. Its `README.md` compares the median runs' throughput, publish p99, end-to-end p99 of the slowest
+  application, and broker CPU per million messages, with the change from A to B, and lists every run with its result
+  and a link to its report, or to its console log when it failed. Beside it are the [comparison charts](#comparison-charts) of the median runs, labeled with the images'
+  tags, or `checkout`.
+
+Prepare the host as for any comparison, and keep the comparison's runs and conclusions together, as the next sections
+describe.
+
 ## Comparing with a released Pulsar
 
 To compare a released Pulsar with the checkout, run the baseline with `-Pperformance.clusterPulsarImage=<image>`,
diff --git a/tests/performance/docs/running-scenarios.md b/tests/performance/docs/running-scenarios.md
index 8232dbf6f093..15035cf72a58 100644
--- a/tests/performance/docs/running-scenarios.md
+++ b/tests/performance/docs/running-scenarios.md
@@ -76,6 +76,8 @@ Pass these with `-P` on the command line, or set them in `~/.gradle/gradle.prope
 | `performance.reportsServer.baseUrl` | The URL of the reports server, such as `http://192.168.1.123:8000/`, with which the launcher prints each report's URL. Default: `http://<bind address>:<port>/`, see [Browsing the reports over HTTP](run-reports.md#browsing-the-reports-over-http). |
 | `performance.profile.maxHeapSize` | The heap of the `profile`, `runJonoffcpuCorrelator` and `runJfrConverter` tasks. Default: `4g`. |
 | `performance.clusterPulsarImage` | A released Pulsar image for the cluster, such as `apachepulsar/pulsar:4.0.13` or `apachepulsar/pulsar:latest`, to test that release instead of the checkout. The tasks build the test image on it, which needs an Alpine-based Pulsar image, and use it for ZooKeeper, the bookies and the brokers; the workloads, and so the Pulsar client, stay on the checkout's image. See [Comparing with a released Pulsar](comparing-revisions.md#comparing-with-a-released-pulsar). |
+| `performance.compare.baselineImage`, `performance.compare.candidateImage` | For `compareImages` and `compareImageWithCheckout`, the Pulsar images that the comparison's clusters run, see [Comparing two Pulsar images](comparing-revisions.md#comparing-two-pulsar-images). `compareImageWithCheckout` takes only the baseline image. |
+| `performance.compare.repetitions`, `performance.compare.medianBy` | For the comparison tasks, the runs per side, 3 by default, and the measure that picks each side's median run: `throughput` (default), `publish-p99`, `e2e-p99` or `broker-cpu` |
 | `docker.tag` | The tag of the Docker images the tasks build and run, `latest` by default. Separate tags keep the images of two revisions apart, see [Comparing revisions](comparing-revisions.md). |
 | `docker.organization` | The organization of the Docker images, `apachepulsar` by default. |
 | `inttest.testImageVariant` | `alpine` (the default, as the unprofiled runs and Pulsar's default image) or `wolfi`, the glibc-based image: the image that profiled runs use, see [Profiling](profiling.md#requirements). |
@@ -113,6 +115,9 @@ Every run gets a directory of its own, in a hierarchy by day, git branch and nam
 - The run directory is named by the run's start in local time. Two runs of the same name started within the same
   second would share it.
 - `--output <dir>` writes the run to exactly that directory instead, outside the hierarchy.
+- The `compareImages` and `compareImageWithCheckout` tasks write their runs as above, and the comparison of them to
+  `<reports root>/<yyyy-MM-dd>/comparisons/<name>/<MM-dd-HH-mm-ss>/`, see
+  [Comparing two Pulsar images](comparing-revisions.md#comparing-two-pulsar-images).
 
 The launcher prints the run directory when it starts, and the run report when it has finished.
 [Finding the results of a run](run-reports.md#finding-the-results-of-a-run) describes finding a run later, and
diff --git a/tests/performance/launcher/build.gradle.kts b/tests/performance/launcher/build.gradle.kts
index 1f643f77e399..f92845abf136 100644
--- a/tests/performance/launcher/build.gradle.kts
+++ b/tests/performance/launcher/build.gradle.kts
@@ -80,7 +80,7 @@ val wolfiTestImage = providers.gradleProperty("inttest.testImageVariant").map {
     }
 }.getOrElse(false)
 
-fun JavaExec.configurePerformanceLauncher(profiler: Boolean) {
+fun JavaExec.configurePerformanceLauncher(profiler: Boolean, clusterPulsarImage: Boolean = true) {
     workingDir(rootProject.projectDir)
     dependsOn(":tests:performance:tools:installDist")
     val imageSuffix = if (profiler && wolfiTestImage) "-wolfi" else ""
@@ -137,7 +137,8 @@ fun JavaExec.configurePerformanceLauncher(profiler: Boolean) {
     // The cluster runs a released Pulsar, -Pperformance.clusterPulsarImage=<image> such as apachepulsar/pulsar:4.0.13,
     // in a test image that :tests:java-test-image:dockerBuildCluster builds on it, with the same tag: ZooKeeper, the
     // bookies and the brokers. The workloads keep this repository's test image, and with it its Pulsar client.
-    providers.gradleProperty("performance.clusterPulsarImage").orNull?.let {
+    // A comparison sets each side's cluster image itself, so it doesn't build this one.
+    providers.gradleProperty("performance.clusterPulsarImage").orNull?.takeIf { clusterPulsarImage }?.let {
         dependsOn(":tests:java-test-image:dockerBuildCluster")
         systemProperty("performance.cluster.pulsarImage", it)
         systemProperty("performance.cluster.image",
@@ -151,6 +152,60 @@ tasks.named<JavaExec>("run") {
     dependsOn(":tests:java-test-image:dockerBuild")
 }
 
+// Compares two Pulsar clusters on a scenario, see tests/performance/docs/comparing-revisions.md: the runs alternate
+// between the sides, -Pperformance.compare.repetitions times each (3 by default), and the comparison charts each side's
+// median run by -Pperformance.compare.medianBy (throughput by default). The gateways and the applications run this
+// checkout's test image, and with it its Pulsar client.
+fun JavaExec.configureImageComparison(candidateIsCheckout: Boolean) {
+    group = "verification"
+    classpath = sourceSets.main.get().runtimeClasspath
+    mainClass.set("org.apache.pulsar.tests.performance.launcher.ImageComparison")
+    configurePerformanceLauncher(profiler = false, clusterPulsarImage = false)
+    dependsOn(":tests:java-test-image:dockerBuild")
+    if (candidateIsCheckout && providers.gradleProperty("performance.compare.candidateImage").isPresent) {
+        doFirst {
+            throw GradleException("compareImageWithCheckout compares with this checkout; use compareImages to compare" +
+                " with -Pperformance.compare.candidateImage")
+        }
+    }
+    val organization = providers.gradleProperty("docker.organization").getOrElse("apachepulsar")
+    fun clusterImage(image: String) =
+        "$organization/java-test-image:cluster-" + image.replace(Regex("[^A-Za-z0-9_.-]"), "-").takeLast(120)
+    val sides = if (candidateIsCheckout) listOf("Baseline") else listOf("Baseline", "Candidate")
+    sides.forEach { side ->
+        val property = "performance.compare.${side.lowercase()}Image"
+        val image = providers.gradleProperty(property).orNull
+        if (image == null) {
+            // Checked when the task runs, so that configuring other tasks doesn't need the property
+            doFirst {
+                throw GradleException("Set -P$property, such as apachepulsar/pulsar:4.0.14")
+            }
+        } else {
+            dependsOn(":tests:java-test-image:dockerBuildCompare$side")
+            systemProperty("performance.compare.${side.lowercase()}.pulsarImage", image)
+            systemProperty("performance.compare.${side.lowercase()}.image", clusterImage(image))
+        }
+    }
+    listOf("repetitions", "medianBy").forEach { name ->
+        providers.gradleProperty("performance.compare.$name").orNull?.let {
+            systemProperty("performance.compare.$name", it)
+        }
+    }
+    outputs.upToDateWhen { false }
+}
+
+tasks.register<JavaExec>("compareImages") {
+    description = "Compare two released Pulsar images on a scenario, -Pperformance.compare.baselineImage and " +
+        "-Pperformance.compare.candidateImage"
+    configureImageComparison(candidateIsCheckout = false)
+}
+
+tasks.register<JavaExec>("compareImageWithCheckout") {
+    description = "Compare a released Pulsar image, -Pperformance.compare.baselineImage, with this checkout on a " +
+        "scenario"
+    configureImageComparison(candidateIsCheckout = true)
+}
+
 tasks.register<JavaExec>("profile") {
     group = "verification"
     description = "Run a standalone performance scenario, profiling the JVMs that have profiler options with " +
diff --git a/tests/performance/launcher/src/main/java/org/apache/pulsar/tests/performance/launcher/ImageComparison.java b/tests/performance/launcher/src/main/java/org/apache/pulsar/tests/performance/launcher/ImageComparison.java
new file mode 100644
index 000000000000..27b9f313d09a
--- /dev/null
+++ b/tests/performance/launcher/src/main/java/org/apache/pulsar/tests/performance/launcher/ImageComparison.java
@@ -0,0 +1,543 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.tests.performance.launcher;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.time.ZonedDateTime;
+import java.time.format.DateTimeFormatter;
+import java.time.temporal.ChronoUnit;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.function.ToDoubleFunction;
+import java.util.stream.Stream;
+import org.apache.pulsar.tests.performance.report.ComparisonCharts;
+import org.apache.pulsar.tests.performance.report.HdrHistogramRenderer;
+import org.apache.pulsar.tests.performance.report.MarkdownPages;
+
+/**
+ * Compares two Pulsar clusters on the same scenario: two released Pulsar images, or a released Pulsar image and this
+ * checkout. It runs the scenario on each side in turn, alternating the order (A B, B A, A B, ...) so that a drift of
+ * the host affects both sides alike, picks each side's median run, charts the two median runs with
+ * {@link ComparisonCharts}, and writes a summary of every run's main measures.
+ *
+ * <p>Each run is a {@link PerformanceLauncher} in a JVM of its own, with this JVM's classpath and its
+ * {@code performance.*} system properties. A side with a released Pulsar image runs ZooKeeper, the bookies and the
+ * brokers in the test image built on it, as {@code -Pperformance.clusterPulsarImage} does; the gateways and the
+ * applications always run this checkout's test image, and with it its Pulsar client. The arguments are the
+ * launcher's, such as {@code --scenario} and {@code --name}.
+ *
+ * <p>System properties, which {@code :tests:performance:launcher:compareImages} and {@code compareImageWithCheckout}
+ * set from Gradle properties:
+ * <ul>
+ *   <li>{@code performance.compare.baseline.pulsarImage} and {@code .image}: the baseline's (A's) released Pulsar
+ *   image and the test image built on it</li>
+ *   <li>{@code performance.compare.candidate.pulsarImage} and {@code .image}: the same for the candidate (B); without
+ *   them, B is this checkout</li>
+ *   <li>{@code performance.compare.repetitions}: the runs per side, 3 by default</li>
+ *   <li>{@code performance.compare.medianBy}: the measure that picks each side's median run, one of
+ *   {@link Measure}'s names, {@code throughput} by default</li>
+ * </ul>
+ */
+public final class ImageComparison {
+    static final String RUN_DIRECTORY_PREFIX = "Run directory: ";
+    static final String SUMMARY_FILE = "README.md";
+    private static final DateTimeFormatter DAY = DateTimeFormatter.ofPattern("yyyy-MM-dd");
+    private static final DateTimeFormatter TIME = DateTimeFormatter.ofPattern("MM-dd-HH-mm-ss");
+    private static final ObjectMapper MAPPER = new ObjectMapper();
+    /** The launcher that runs now, which a shutdown hook stops when the comparison is interrupted. */
+    private static volatile Process current;
+
+    /** A side of the comparison: a released Pulsar image, or this checkout when {@code pulsarImage} is null. */
+    record Side(String name, String pulsarImage, String clusterImage) {
+        boolean checkout() {
+            return pulsarImage == null;
+        }
+
+        /** The side's name in the charts and the summary: the image's tag, or {@code checkout}. */
+        String label() {
+            if (checkout()) {
+                return "checkout";
+            }
+            int slash = pulsarImage.lastIndexOf('/');
+            int colon = pulsarImage.lastIndexOf(':');
+            return colon > slash ? pulsarImage.substring(colon + 1) : pulsarImage.substring(slash + 1);
+        }
+
+        String description() {
+            return checkout() ? "this checkout" : "`" + pulsarImage + "`";
+        }
+    }
+
+    /** A run's main measures; NaN when its outputs don't have one. */
+    record Measures(double throughput, double publishP99Millis, double endToEndP99Millis,
+                    double brokerCpuSecondsPerMillion) {
+    }
+
+    /** The measures that the summary compares, and that can pick each side's median run. */
+    enum Measure {
+        THROUGHPUT("throughput", "throughput", "Throughput, msg/s", Measures::throughput, "%,.0f"),
+        PUBLISH_P99("publish-p99", "publish p99", "Publish p99, ms", Measures::publishP99Millis, "%,.1f"),
+        END_TO_END_P99("e2e-p99", "end-to-end p99", "End-to-end p99 (slowest application), ms",
+                Measures::endToEndP99Millis, "%,.1f"),
+        BROKER_CPU("broker-cpu", "broker CPU per message", "Broker CPU per million messages, s",
+                Measures::brokerCpuSecondsPerMillion, "%,.1f");
+
+        final String option;
+        final String name;
+        final String title;
+        final ToDoubleFunction<Measures> value;
+        final String format;
+
+        Measure(String option, String name, String title, ToDoubleFunction<Measures> value, String format) {
+            this.option = option;
+            this.name = name;
+            this.title = title;
+            this.value = value;
+            this.format = format;
+        }
+
+        static Measure of(String option) {
+            for (Measure measure : values()) {
+                if (measure.option.equals(option)) {
+                    return measure;
+                }
+            }
+            throw new IllegalArgumentException("performance.compare.medianBy must be one of throughput, publish-p99,"
+                    + " e2e-p99 or broker-cpu, not '" + option + "'");
+        }
+
+        String format(double v) {
+            return Double.isNaN(v) ? "–" : String.format(Locale.ROOT, format, v);
+        }
+    }
+
+    /** How a run ended. Only valid runs count in the comparison. */
+    enum Result {
+        VALID("ok"),
+        FAILED("failed"),
+        /** The applications received messages out of order or invalid messages, so the run measured a bug. */
+        INVALID("invalid: ordering violations or invalid messages");
+
+        final String text;
+
+        Result(String text) {
+            this.text = text;
+        }
+    }
+
+    /** A run of the comparison: its side, its run directory, how it ended, and its measures. */
+    record Run(Side side, int order, Path directory, Result result, Measures measures) {
+        boolean valid() {
+            return result == Result.VALID;
+        }
+    }
+
+    private static final Measures NO_MEASURES = new Measures(Double.NaN, Double.NaN, Double.NaN, Double.NaN);
+
+    private ImageComparison() {
+    }
+
+    public static void main(String[] args) throws Exception {
+        System.exit(run(args));
+    }
+
+    static int run(String[] launcherArgs) throws Exception {
+        for (String arg : launcherArgs) {
+            if (arg.equals("--output") || arg.startsWith("--output=")) {
+                throw new IllegalArgumentException("A comparison writes each run to its own directory; don't pass"
+                        + " --output");
+            }
+        }
+        Side baseline = side("A", "baseline");
+        if (baseline == null) {
+            throw new IllegalArgumentException("Set performance.compare.baseline.pulsarImage, such as with"
+                    + " -Pperformance.compare.baselineImage=apachepulsar/pulsar:4.0.14");
+        }
+        Side candidate = side("B", "candidate");
+        if (candidate == null) {
+            candidate = new Side("B", null, null);
+        }
+        int repetitions = Integer.parseInt(System.getProperty("performance.compare.repetitions", "3"));
+        if (repetitions < 1) {
+            throw new IllegalArgumentException("performance.compare.repetitions must be at least 1");
+        }
+        Measure medianBy = Measure.of(System.getProperty("performance.compare.medianBy", "throughput"));
+        ZonedDateTime started = ZonedDateTime.now().truncatedTo(ChronoUnit.SECONDS);
+
+        // Stop the launcher that runs when the comparison is interrupted, such as by Ctrl-C or a cancelled Gradle
+        // build, so that it stops its containers instead of running on without anyone waiting for it
+        Runtime.getRuntime().addShutdownHook(new Thread(ImageComparison::stopCurrent, "stop-comparison-run"));
+
+        List<Run> runs = new ArrayList<>();
+        List<Side> order = runOrder(baseline, candidate, repetitions);
+        for (int i = 0; i < order.size(); i++) {
+            Side side = order.get(i);
+            System.out.printf("=== Comparison run %d of %d: %s (%s)%n", i + 1, order.size(), side.name(),
+                    side.checkout() ? "this checkout" : side.pulsarImage());
+            Run run = launch(side, i + 1, launcherArgs);
+            runs.add(run);
+            if (run.directory() == null) {
+                // The launcher failed before it started the run, such as on an invalid argument, which fails the other
+                // runs alike
+                System.err.println("The launcher failed before it started run " + run.order()
+                        + "; the comparison stops");
+                return 1;
+            }
+        }
+
+        Path firstRun = runs.get(0).directory();
+        Path output = outputDirectory(firstRun, started);
+        Files.createDirectories(output);
+        Run medianA = median(runs, baseline, medianBy);
+        Run medianB = median(runs, candidate, medianBy);
+        List<String> charts = List.of();
+        boolean charted = false;
+        if (medianA != null && medianB != null) {
+            String labelA = baseline.label();
+            String labelB = comparisonLabel(baseline, candidate);
+            int chartsExit = ComparisonCharts.execute("--baseline",
+                    medianA.directory().toString(), "--comparison", medianB.directory().toString(),
+                    "--baseline-label", labelA, "--comparison-label", labelB, "--output", output.toString());
+            charted = chartsExit == 0;
+            if (!charted) {
+                System.err.println("Charting the median runs failed");
+            }
+            try (Stream<Path> files = Files.list(output)) {
+                charts = files.map(p -> p.getFileName().toString())
+                        .filter(f -> f.endsWith(".svg") && !f.endsWith("-separate.svg"))
+                        .sorted(Comparator.comparingInt(ImageComparison::chartRank)
+                                .thenComparing(Comparator.naturalOrder()))
+                        .toList();
+            }
+        }
+        Path summary = output.resolve(SUMMARY_FILE);
+        Files.writeString(summary, summary(baseline, candidate, launcherArgs, repetitions, medianBy, runs, medianA,
+                medianB, charts, output));
+        // index.html beside it, with the links to the runs' reports pointing to their pages, as the runs have
+        MarkdownPages.renderHtml(summary, output, "Comparison of " + baseline.label() + " and " + candidate.label());
+        System.out.println("Comparison: " + output.resolve(SUMMARY_FILE));
+        if (medianA == null || medianB == null) {
+            System.err.println("A side has no valid run with the measure " + medianBy.name + " to compare");
+        }
+        return charted ? 0 : 1;
+    }
+
+    private static void stopCurrent() {
+        Process process = current;
+        if (process == null || !process.isAlive()) {
+            return;
+        }
+        // The launcher's own shutdown hook stops the run's containers, which takes a while
+        process.destroy();
+        try {
+            if (!process.waitFor(60, TimeUnit.SECONDS)) {
+                process.destroyForcibly();
+            }
+        } catch (InterruptedException e) {
+            process.destroyForcibly();
+            Thread.currentThread().interrupt();
+        }
+    }
+
+    private static Side side(String name, String key) {
+        String pulsarImage = System.getProperty("performance.compare." + key + ".pulsarImage");
+        String clusterImage = System.getProperty("performance.compare." + key + ".image");
+        if ((pulsarImage == null) != (clusterImage == null)) {
+            throw new IllegalArgumentException("Set both performance.compare." + key + ".pulsarImage and .image, or"
+                    + " neither; the compareImages and compareImageWithCheckout tasks set both");
+        }
+        return pulsarImage == null ? null : new Side(name, pulsarImage, clusterImage);
+    }
+
+    /** The sides in the order to run them: A B, then B A, and so on, so that each side runs first as often. */
+    static List<Side> runOrder(Side a, Side b, int repetitions) {
+        List<Side> order = new ArrayList<>();
+        for (int i = 0; i < repetitions; i++) {
+            order.add(i % 2 == 0 ? a : b);
+            order.add(i % 2 == 0 ? b : a);
+        }
+        return order;
+    }
+
+    /** B's label in the charts, which tells it apart from A's when the images have the same tag. */
+    static String comparisonLabel(Side a, Side b) {
+        return b.label().equals(a.label()) ? b.label() + "-B" : b.label();
+    }
+
+    /** The charts in the order of a run's report: throughput, latency, then backlog. */
+    static int chartRank(String chart) {
+        if (chart.startsWith("throughput")) {
+            return 0;
+        } else if (chart.startsWith("latency-percentiles-log")) {
+            return 2;
+        } else if (chart.startsWith("latency-percentiles")) {
+            return 1;
+        } else if (chart.startsWith("backlog")) {
+            return 3;
+        }
+        return 4;
+    }
+
+    /** The JVM command line that runs the launcher for a side, with this JVM's performance.* properties. */
+    static List<String> command(Side side, String javaHome, String classpath, Map<Object, Object> properties,
+                                String[] launcherArgs) {
+        List<String> command = new ArrayList<>();
+        command.add(Path.of(javaHome, "bin", "java").toString());
+        properties.entrySet().stream()
+                .filter(e -> e.getKey() instanceof String key && key.startsWith("performance.")
+                        && !key.startsWith("performance.compare.") && !key.startsWith("performance.cluster."))
+                .sorted(Comparator.comparing(e -> (String) e.getKey()))
+                .forEach(e -> command.add("-D" + e.getKey() + "=" + e.getValue()));
+        if (!side.checkout()) {
+            command.add("-Dperformance.cluster.pulsarImage=" + side.pulsarImage());
+            command.add("-Dperformance.cluster.image=" + side.clusterImage());
+        }
+        command.add("-cp");
+        command.add(classpath);
+        command.add(PerformanceLauncher.class.getName());
+        command.addAll(List.of(launcherArgs));
+        return command;
+    }
+
+    private static Run launch(Side side, int order, String[] launcherArgs) throws IOException, InterruptedException {
+        ProcessBuilder builder = new ProcessBuilder(command(side, System.getProperty("java.home"),
+                System.getProperty("java.class.path"), System.getProperties(), launcherArgs))
+                .redirectErrorStream(true);
+        Process process = builder.start();
+        current = process;
+        Path directory = null;
+        try (BufferedReader output = new BufferedReader(
+                new InputStreamReader(process.getInputStream(), StandardCharsets.UTF_8))) {
+            String line;
+            while ((line = output.readLine()) != null) {
+                System.out.println(line);
+                if (directory == null && line.startsWith(RUN_DIRECTORY_PREFIX)) {
+                    directory = Path.of(line.substring(RUN_DIRECTORY_PREFIX.length()).trim());
+                }
+            }
+        } catch (IOException e) {
+            process.destroy();
+            throw e;
+        }
+        int exit;
+        try {
+            exit = process.waitFor();
+        } catch (InterruptedException e) {
+            process.destroy();
+            throw e;
+        } finally {
+            current = null;
+        }
+        if (exit != 0 || directory == null) {
+            System.out.printf("=== Run %d of %s failed with exit code %d; the comparison leaves it out%n", order,
+                    side.name(), exit);
+            return new Run(side, order, directory, Result.FAILED, NO_MEASURES);
+        }
+        Result result = Result.VALID;
+        if (deliveredInvalidly(directory)) {
+            System.out.printf("=== Run %d of %s received messages out of order or invalid messages; the comparison"
+                    + " leaves it out%n", order, side.name());
+            result = Result.INVALID;
+        }
+        return new Run(side, order, directory, result, measures(directory));
+    }
+
+    /** Whether an application of the run received messages out of order or invalid messages. */
+    static boolean deliveredInvalidly(Path run) {
+        try (Stream<Path> summaries = applicationFiles(run, "application-summary.json")) {
+            for (Iterator<Path> it = summaries.iterator(); it.hasNext(); ) {
+                JsonNode summary = MAPPER.readTree(it.next().toFile());
+                if (summary.path("orderingViolations").asLong() > 0 || summary.path("invalidMessages").asLong() > 0) {
+                    return true;
+                }
+            }
+            return false;
+        } catch (IOException e) {
+            System.err.println("Reading the applications' summaries of " + run + " failed: " + e);
+            return true;
+        }
+    }
+
+    /** A run's measures from its outputs; a measure is NaN when its outputs are missing or can't be read. */
+    static Measures measures(Path run) {
+        return new Measures(measure(run, "throughput", () -> {
+            JsonNode summary = MAPPER.readTree(run.resolve("gateways").resolve("gateways-summary.json").toFile());
+            return summary.path("messagesPerSecond").asDouble(Double.NaN);
+        }), measure(run, "publish p99", () -> p99Millis(List.of(run.resolve("gateways")
+                .resolve("gateways-latency.hdr")))), measure(run, "end-to-end p99", () -> {
+            double slowest = Double.NaN;
+            try (Stream<Path> logs = applicationFiles(run, "application-latency.hdr")) {
+                for (Iterator<Path> it = logs.iterator(); it.hasNext(); ) {
+                    double p99 = p99Millis(List.of(it.next()));
+                    slowest = Double.isNaN(slowest) ? p99 : Math.max(slowest, p99);
+                }
+            }
+            return slowest;
+        }), measure(run, "broker CPU", () -> {
+            double brokerCpu = Double.NaN;
+            for (Map.Entry<String, JsonNode> e : MAPPER.readTree(run.resolve("container-summary.json").toFile())
+                    .path("containers").properties()) {
+                double v = e.getValue().path("cpuSecondsPerMillionMessages").asDouble(Double.NaN);
+                if (e.getKey().startsWith("broker") && !Double.isNaN(v)) {
+                    brokerCpu = Double.isNaN(brokerCpu) ? v : brokerCpu + v;
+                }
+            }
+            return brokerCpu;
+        }));
+    }
+
+    private interface MeasureReader {
+        double read() throws IOException;
+    }
+
+    private static double measure(Path run, String name, MeasureReader reader) {
+        try {
+            return reader.read();
+        } catch (IOException | RuntimeException e) {
+            System.err.println("Reading the " + name + " of " + run + " failed: " + e);
+            return Double.NaN;
+        }
+    }
+
+    /** The files of that name in the run's applications' directories. */
+    private static Stream<Path> applicationFiles(Path run, String name) throws IOException {
+        Path applications = run.resolve("applications");
+        if (!Files.isDirectory(applications)) {
+            return Stream.empty();
+        }
+        return Files.list(applications).map(dir -> dir.resolve(name)).filter(Files::isRegularFile).sorted();
+    }
+
+    /** The 99th percentile of HdrHistogram latency logs, which record microseconds, in milliseconds. */
+    static double p99Millis(List<Path> logs) throws IOException {
+        return HdrHistogramRenderer.valueAtPercentileMillis(logs, 99.0);
+    }
+
+    /**
+     * The side's median valid run by the measure, the lower one of the middle two for an even count, leaving out the
+     * runs without the measure; null when no run has it.
+     */
+    static Run median(List<Run> runs, Side side, Measure by) {
+        List<Run> valid = runs.stream()
+                .filter(r -> r.side() == side && r.valid() && !Double.isNaN(by.value.applyAsDouble(r.measures())))
+                .sorted(Comparator.comparingDouble(r -> by.value.applyAsDouble(r.measures()))).toList();
+        return valid.isEmpty() ? null : valid.get((valid.size() - 1) / 2);
+    }
+
+    /** {@code <reports root>/<day>/comparisons/<run name>/<time>}, beside the runs' days in the reports root. */
+    static Path outputDirectory(Path firstRun, ZonedDateTime started) {
+        // <reports root>/<day>/<branch or image>/<name>/<time>
+        Path name = firstRun.getParent();
+        Path reportsRoot = name.getParent().getParent().getParent();
+        return reportsRoot.resolve(DAY.format(started)).resolve("comparisons").resolve(name.getFileName())
+                .resolve(TIME.format(started));
+    }
+
+    static String summary(Side a, Side b, String[] launcherArgs, int repetitions, Measure medianBy, List<Run> runs,
+                          Run medianA, Run medianB, List<String> charts, Path output) {
+        StringBuilder s = new StringBuilder();
+        s.append("# Comparison: ").append(a.label()).append(" (A) and ").append(b.label()).append(" (B)\n\n");
+        s.append("- **A:** ").append(a.description()).append("\n");
+        s.append("- **B:** ").append(b.description()).append("\n");
+        s.append("- **Clusters:** ZooKeeper, the bookies and the brokers run each side's Pulsar. The gateways and the"
+                + " applications run this checkout's test image, and with it its Pulsar client.\n");
+        s.append("- **Runs:** ").append(repetitions).append(" per side, alternating the order (A B, B A, ...): `")
+                .append(String.join(" ", launcherArgs)).append("`\n");
+        s.append("- **Median run:** each side's median valid run by ").append(medianBy.name)
+                .append(", the lower of the middle two for an even count, which the charts compare. A run that failed,"
+                        + " or whose applications received messages out of order or invalid messages, doesn't"
+                        + " count.\n\n");
+
+        s.append("| Median run | A | B | Change |\n|---|---:|---:|---:|\n");
+        for (Measure measure : Measure.values()) {
+            double va = medianA != null ? measure.value.applyAsDouble(medianA.measures()) : Double.NaN;
+            double vb = medianB != null ? measure.value.applyAsDouble(medianB.measures()) : Double.NaN;
+            String change = Double.isNaN(va) || Double.isNaN(vb) || va == 0 ? "–"
+                    : String.format(Locale.ROOT, "%+.1f %%", (vb - va) * 100 / va);
+            s.append("| ").append(measure.title).append(" | ").append(measure.format(va)).append(" | ")
+                    .append(measure.format(vb)).append(" | ").append(change).append(" |\n");
+        }
+
+        s.append("\n## Runs\n\n| # | Side | Run | Result |");
+        for (Measure measure : Measure.values()) {
+            s.append(' ').append(measure.title).append(" |");
+        }
+        s.append("\n|---:|---|---|---|");
+        s.append("---:|".repeat(Measure.values().length)).append('\n');
+        for (Run run : runs) {
+            String result = run == medianA || run == medianB ? "median" : run.result().text;
+            s.append("| ").append(run.order()).append(" | ").append(run.side().name()).append(" | ")
+                    .append(runLink(run, output))
+                    .append(" | ").append(result).append(" |");
+            for (Measure measure : Measure.values()) {
+                s.append(' ').append(measure.format(measure.value.applyAsDouble(run.measures()))).append(" |");
+            }
+            s.append('\n');
+        }
+        if (!charts.isEmpty()) {
+            s.append("\n## The median runs\n\n");
+            String labels = a.label() + " and " + comparisonLabel(a, b);
+            for (String chart : charts) {
+                s.append("![").append(chartTitle(chart)).append(" of ").append(labels).append("](").append(chart)
+                        .append(")\n\n");
+            }
+        }
+        return s.toString();
+    }
+
+    /**
+     * A link to the run's report, or to its console log when it failed, since a failed run may have no report; a dash
+     * when the launcher didn't start the run.
+     */
+    private static String runLink(Run run, Path output) {
+        if (run.directory() == null) {
+            return "–";
+        }
+        boolean failed = run.result() == Result.FAILED;
+        Path target = run.directory().resolve(failed ? "console.log.txt" : SUMMARY_FILE);
+        return "[" + (failed ? "Console log" : "Report") + " of run " + run.order() + " (" + run.side().name()
+                + ")](" + output.relativize(target).toString().replace('\\', '/') + ")";
+    }
+
+    /** What a chart of {@link ComparisonCharts} shows, from its file name. */
+    static String chartTitle(String chart) {
+        if (chart.startsWith("throughput")) {
+            return "Throughput";
+        } else if (chart.startsWith("latency-percentiles-log")) {
+            return "Latency percentiles on a logarithmic scale";
+        } else if (chart.startsWith("latency-percentiles")) {
+            return "Latency percentiles";
+        } else if (chart.startsWith("backlog")) {
+            return "Backlog";
+        }
+        return chart.substring(0, chart.length() - ".svg".length());
+    }
+}
diff --git a/tests/performance/launcher/src/test/java/org/apache/pulsar/tests/performance/launcher/ImageComparisonTest.java b/tests/performance/launcher/src/test/java/org/apache/pulsar/tests/performance/launcher/ImageComparisonTest.java
new file mode 100644
index 000000000000..0b1cf216b71e
--- /dev/null
+++ b/tests/performance/launcher/src/test/java/org/apache/pulsar/tests/performance/launcher/ImageComparisonTest.java
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.tests.performance.launcher;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.assertj.core.api.Assertions.within;
+import java.io.IOException;
+import java.io.PrintStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.time.ZonedDateTime;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import org.HdrHistogram.Histogram;
+import org.HdrHistogram.HistogramLogWriter;
+import org.apache.pulsar.tests.performance.launcher.ImageComparison.Measure;
+import org.apache.pulsar.tests.performance.launcher.ImageComparison.Measures;
+import org.apache.pulsar.tests.performance.launcher.ImageComparison.Result;
+import org.apache.pulsar.tests.performance.launcher.ImageComparison.Run;
+import org.apache.pulsar.tests.performance.launcher.ImageComparison.Side;
+import org.testng.annotations.Test;
+
+public class ImageComparisonTest {
+    private static final Side A = new Side("A", "apachepulsar/pulsar:4.0.14", "apachepulsar/java-test-image:cluster-a");
+    private static final Side B = new Side("B", "apachepulsar/pulsar:5.0.0", "apachepulsar/java-test-image:cluster-b");
+    private static final Side CHECKOUT = new Side("B", null, null);
+
+    @Test
+    public void alternatesWhichSideRunsFirst() {
+        assertThat(ImageComparison.runOrder(A, B, 3)).containsExactly(A, B, B, A, A, B);
+    }
+
+    @Test
+    public void labelsASideByItsImageTag() {
+        assertThat(A.label()).isEqualTo("4.0.14");
+        assertThat(new Side("A", "localhost:5000/pulsar", "x").label()).isEqualTo("pulsar");
+        assertThat(CHECKOUT.label()).isEqualTo("checkout");
+    }
+
+    @Test
+    public void runsTheLauncherWithTheSidesClusterImageAndThePerformanceProperties() {
+        Map<Object, Object> properties = Map.of(
+                "performance.reports.dir", "/reports",
+                "performance.compare.repetitions", "3",
+                "performance.cluster.pulsarImage", "ignored",
+                "user.home", "/home/u");
+        String[] launcherArgs = {"--scenario", "s.yaml", "--name", "ab"};
+
+        assertThat(ImageComparison.command(A, "/jdk", "cp", properties, launcherArgs)).containsExactly(
+                Path.of("/jdk", "bin", "java").toString(),
+                "-Dperformance.reports.dir=/reports",
+                "-Dperformance.cluster.pulsarImage=apachepulsar/pulsar:4.0.14",
+                "-Dperformance.cluster.image=apachepulsar/java-test-image:cluster-a",
+                "-cp", "cp", PerformanceLauncher.class.getName(), "--scenario", "s.yaml", "--name", "ab");
+        // the checkout runs its own test image for the cluster too
+        assertThat(ImageComparison.command(CHECKOUT, "/jdk", "cp", properties, launcherArgs))
+                .doesNotContain("-Dperformance.cluster.pulsarImage=apachepulsar/pulsar:4.0.14")
+                .noneMatch(arg -> arg.startsWith("-Dperformance.cluster."));
+    }
+
+    @Test
+    public void readsARunsMeasures() throws IOException {
+        Path run = Files.createTempDirectory("image-comparison");
+        Files.createDirectories(run.resolve("gateways"));
+        Files.writeString(run.resolve("gateways/gateways-summary.json"), "{\"messagesPerSecond\": 110548.4}");
+        // 1,000 latencies of 1 to 1,000 ms, so that the 99th percentile is 990 ms
+        writeLatencies(run.resolve("gateways/gateways-latency.hdr"), 1);
+        for (String application : List.of("iot-application-0", "iot-application-1")) {
+            Files.createDirectories(run.resolve("applications").resolve(application));
+        }
+        writeLatencies(run.resolve("applications/iot-application-0/application-latency.hdr"), 2);
+        writeLatencies(run.resolve("applications/iot-application-1/application-latency.hdr"), 3);
+        Files.writeString(run.resolve("container-summary.json"), "{\"containers\": {"
+                + "\"broker-0\": {\"cpuSecondsPerMillionMessages\": 40.0},"
+                + "\"broker-1\": {\"cpuSecondsPerMillionMessages\": 2.5},"
+                + "\"bookie-0\": {\"cpuSecondsPerMillionMessages\": 30.0}}}");
+
+        Measures measures = ImageComparison.measures(run);
+        assertThat(measures.throughput()).isEqualTo(110548.4);
+        assertThat(measures.publishP99Millis()).isCloseTo(990, within(1.0));
+        // the slowest application's
+        assertThat(measures.endToEndP99Millis()).isCloseTo(2970, within(3.0));
+        assertThat(measures.brokerCpuSecondsPerMillion()).isEqualTo(42.5);
+    }
+
+    @Test
+    public void aMeasureWithoutItsOutputsIsNaN() throws IOException {
+        Path run = Files.createTempDirectory("image-comparison");
+        Files.createDirectories(run.resolve("gateways"));
+        Files.writeString(run.resolve("gateways/gateways-summary.json"), "not json");
+
+        PrintStream err = System.err;
+        try {
+            System.setErr(new PrintStream(PrintStream.nullOutputStream()));
+            assertThat(ImageComparison.measures(run))
+                    .isEqualTo(new Measures(Double.NaN, Double.NaN, Double.NaN, Double.NaN));
+        } finally {
+            System.setErr(err);
+        }
+    }
+
+    // a latency log as the gateways and the applications write it, in microseconds: 1 to 1,000 ms times the factor
+    private static void writeLatencies(Path log, int factor) throws IOException {
+        Histogram histogram = new Histogram(3);
+        for (int millis = 1; millis <= 1000; millis++) {
+            histogram.recordValue(millis * factor * 1000L);
+        }
+        try (PrintStream out = new PrintStream(Files.newOutputStream(log))) {
+            HistogramLogWriter writer = new HistogramLogWriter(out);
+            writer.outputLogFormatVersion();
+            writer.outputLegend();
+            writer.outputIntervalHistogram(histogram);
+        }
+    }
+
+    @Test
+    public void recognizesARunWhoseApplicationsReceivedMessagesOutOfOrder() throws IOException {
+        Path run = Files.createTempDirectory("image-comparison");
+        for (String application : List.of("iot-application-0", "iot-application-1")) {
+            Files.createDirectories(run.resolve("applications").resolve(application));
+        }
+        Files.writeString(run.resolve("applications/iot-application-0/application-summary.json"),
+                "{\"duplicates\": 3, \"orderingViolations\": 0, \"invalidMessages\": 0}");
+        Path second = run.resolve("applications/iot-application-1/application-summary.json");
+        Files.writeString(second, "{\"duplicates\": 0, \"orderingViolations\": 0, \"invalidMessages\": 0}");
+        // duplicates are a redelivery, which doesn't make the measures wrong
+        assertThat(ImageComparison.deliveredInvalidly(run)).isFalse();
+
+        Files.writeString(second, "{\"duplicates\": 0, \"orderingViolations\": 2, \"invalidMessages\": 0}");
+        assertThat(ImageComparison.deliveredInvalidly(run)).isTrue();
+        Files.writeString(second, "{\"duplicates\": 0, \"orderingViolations\": 0, \"invalidMessages\": 1}");
+        assertThat(ImageComparison.deliveredInvalidly(run)).isTrue();
+    }
+
+    @Test
+    public void picksTheMedianValidRunOfASide() {
+        Run a1 = run(A, 1, Result.VALID, 100);
+        Run a2 = run(A, 4, Result.VALID, 120);
+        Run a3 = run(A, 5, Result.VALID, 110);
+        Run a4 = run(A, 7, Result.FAILED, 999);
+        Run a5 = run(A, 8, Result.INVALID, 105);
+        Run a6 = run(A, 9, Result.VALID, Double.NaN);
+        Run b1 = run(B, 2, Result.VALID, 200);
+        Run b2 = run(B, 3, Result.VALID, 190);
+        List<Run> runs = List.of(a1, b1, b2, a2, a3, a4, a5, a6);
+
+        // neither the failed and invalid runs nor the run without the measure count
+        assertThat(ImageComparison.median(runs, A, Measure.THROUGHPUT)).isSameAs(a3);
+        // the lower of the middle two
+        assertThat(ImageComparison.median(runs, B, Measure.THROUGHPUT)).isSameAs(b2);
+        assertThat(ImageComparison.median(List.of(a4, a5, a6), A, Measure.THROUGHPUT)).isNull();
+    }
+
+    private static Run run(Side side, int order, Result result, double throughput) {
+        return new Run(side, order, Path.of("/reports/2026-10-06/pulsar-x/ab/10-06-10-00-0" + order), result,
+                new Measures(throughput, 10, 20, 30));
+    }
+
+    @Test
+    public void writesTheComparisonBesideTheRunsDays() {
+        Path firstRun = Path.of("/reports/2026-10-06/pulsar-4.0.14/ab/10-06-10-00-01");
+
+        assertThat(ImageComparison.outputDirectory(firstRun, ZonedDateTime.parse("2026-10-06T09:59:58+03:00")))
+                .isEqualTo(Path.of("/reports/2026-10-06/comparisons/ab/10-06-09-59-58"));
+    }
+
+    @Test
+    public void summarizesTheMedianRunsAndEveryRun() {
+        Measures none = new Measures(Double.NaN, Double.NaN, Double.NaN, Double.NaN);
+        Run a = new Run(A, 1, Path.of("/reports/2026-10-06/pulsar-4.0.14/ab/10-06-10-00-01"), Result.VALID,
+                new Measures(100_000, 10, 20, 40));
+        Run b = new Run(B, 2, Path.of("/reports/2026-10-06/pulsar-5.0.0/ab/10-06-10-05-01"), Result.VALID,
+                new Measures(110_000, 9, 18, 36));
+        Run failed = new Run(B, 3, Path.of("/reports/2026-10-06/pulsar-5.0.0/ab/10-06-10-10-01"), Result.FAILED,
+                none);
+        Run invalid = new Run(A, 4, Path.of("/reports/2026-10-06/pulsar-4.0.14/ab/10-06-10-15-01"), Result.INVALID,
+                new Measures(90_000, 10, 20, 40));
+
+        String summary = ImageComparison.summary(A, B, new String[] {"--scenario", "s.yaml"}, 2,
+                Measure.THROUGHPUT, List.of(a, b, failed, invalid), a, b,
+                List.of("throughput-4.0.14-vs-5.0.0.svg", "latency-percentiles-log-4.0.14-vs-5.0.0.svg"),
+                Path.of("/reports/2026-10-06/comparisons/ab/10-06-09-59-58"));
+
+        assertThat(summary)
+                .contains("# Comparison: 4.0.14 (A) and 5.0.0 (B)")
+                .contains("- **A:** `apachepulsar/pulsar:4.0.14`")
+                .contains("| Throughput, msg/s | 100,000 | 110,000 | +10.0 % |")
+                .contains("| Broker CPU per million messages, s | 40.0 | 36.0 | -10.0 % |")
+                .contains("| 1 | A | [Report of run 1 (A)](../../../pulsar-4.0.14/ab/10-06-10-00-01/README.md) | median"
+                        + " |")
+                .contains("each side's median valid run by throughput, the lower of the middle two for an even count,"
+                        + " which the charts compare")
+                .contains("| 3 | B | [Console log of run 3 (B)]"
+                        + "(../../../pulsar-5.0.0/ab/10-06-10-10-01/console.log.txt) | failed | – | – | – | – |")
+                .contains("| 4 | A | [Report of run 4 (A)](../../../pulsar-4.0.14/ab/10-06-10-15-01/README.md)"
+                        + " | invalid: ordering violations or invalid messages | 90,000 |")
+                .contains("![Throughput of 4.0.14 and 5.0.0](throughput-4.0.14-vs-5.0.0.svg)")
+                .contains("![Latency percentiles on a logarithmic scale of 4.0.14 and 5.0.0]"
+                        + "(latency-percentiles-log-4.0.14-vs-5.0.0.svg)");
+    }
+
+    @Test
+    public void tellsTheSidesApartWhenTheirImagesHaveTheSameTag() {
+        Side mirror = new Side("B", "localhost:5000/pulsar:4.0.14", "x");
+        assertThat(ImageComparison.comparisonLabel(A, mirror)).isEqualTo("4.0.14-B");
+        assertThat(ImageComparison.comparisonLabel(A, B)).isEqualTo("5.0.0");
+    }
+
+    @Test
+    public void ordersTheChartsAsARunsReport() {
+        List<String> charts = new ArrayList<>(List.of("backlog-a-vs-b.svg", "latency-percentiles-log-a-vs-b.svg",
+                "throughput-a-vs-b.svg", "latency-percentiles-a-vs-b.svg"));
+        charts.sort(Comparator.comparingInt(ImageComparison::chartRank));
+        assertThat(charts).containsExactly("throughput-a-vs-b.svg", "latency-percentiles-a-vs-b.svg",
+                "latency-percentiles-log-a-vs-b.svg", "backlog-a-vs-b.svg");
+    }
+
+    @Test
+    public void rejectsAnUnknownMedianMeasure() {
+        assertThat(Measure.of("e2e-p99")).isEqualTo(Measure.END_TO_END_P99);
+        assertThatThrownBy(() -> Measure.of("latency")).hasMessageContaining("throughput, publish-p99");
+    }
+}
diff --git a/tests/performance/report-tool/src/main/java/org/apache/pulsar/tests/performance/report/ComparisonCharts.java b/tests/performance/report-tool/src/main/java/org/apache/pulsar/tests/performance/report/ComparisonCharts.java
index fa9da0d02e40..baeacd5cd92a 100644
--- a/tests/performance/report-tool/src/main/java/org/apache/pulsar/tests/performance/report/ComparisonCharts.java
+++ b/tests/performance/report-tool/src/main/java/org/apache/pulsar/tests/performance/report/ComparisonCharts.java
@@ -107,7 +107,12 @@ public final class ComparisonCharts implements Callable<Integer> {
     }
 
     public static void main(String[] args) {
-        System.exit(new CommandLine(new ComparisonCharts()).execute(args));
+        System.exit(execute(args));
+    }
+
+    /** Charts the runs as the command line does, and returns its exit code instead of exiting. */
+    public static int execute(String... args) {
+        return new CommandLine(new ComparisonCharts()).execute(args);
     }
 
     @Override
diff --git a/tests/performance/report-tool/src/main/java/org/apache/pulsar/tests/performance/report/HdrHistogramRenderer.java b/tests/performance/report-tool/src/main/java/org/apache/pulsar/tests/performance/report/HdrHistogramRenderer.java
index 46a2ca3bd99a..bf65ec3de5ba 100644
--- a/tests/performance/report-tool/src/main/java/org/apache/pulsar/tests/performance/report/HdrHistogramRenderer.java
+++ b/tests/performance/report-tool/src/main/java/org/apache/pulsar/tests/performance/report/HdrHistogramRenderer.java
@@ -223,6 +223,16 @@ public final class HdrHistogramRenderer implements Callable<Integer> {
                 + "%";
     }
 
+    /**
+     * The latency at a percentile of the latency logs, merged, in milliseconds.
+     *
+     * @param percentile such as 99.0
+     * @throws IOException when the logs can't be read or have no observations
+     */
+    public static double valueAtPercentileMillis(List<Path> logs, double percentile) throws IOException {
+        return readMerged(logs).getValueAtPercentile(percentile) / 1000.0;
+    }
+
     static Histogram readMerged(List<Path> paths) throws IOException {
         Histogram merged = null;
         for (Path path : paths) {
diff --git a/tests/performance/report-tool/src/main/java/org/apache/pulsar/tests/performance/report/MarkdownPages.java b/tests/performance/report-tool/src/main/java/org/apache/pulsar/tests/performance/report/MarkdownPages.java
index 6ebf1130609f..7b1d763b70ba 100644
--- a/tests/performance/report-tool/src/main/java/org/apache/pulsar/tests/performance/report/MarkdownPages.java
+++ b/tests/performance/report-tool/src/main/java/org/apache/pulsar/tests/performance/report/MarkdownPages.java
@@ -71,7 +71,7 @@ public final class MarkdownPages {
      * @param root the run directory: absolute paths inside it become relative links
      * @return the HTML page
      */
-    static Path renderHtml(Path markdown, Path root, String title) throws IOException {
+    public static Path renderHtml(Path markdown, Path root, String title) throws IOException {
         return renderHtml(markdown, root, title, false);
     }