You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by lh...@apache.org on 2026/10/04 14:44:46 UTC

(pulsar) branch master updated: [improve][test] Add a catch-up scenario: consumers joining a live stream later (#26826)

This is an automated email from the ASF dual-hosted git repository.

lhotari pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/master by this push:
     new b2f6041b872d [improve][test] Add a catch-up scenario: consumers joining a live stream later (#26826)
b2f6041b872d is described below

commit b2f6041b872de8940521d83cad6f98f49608c42c
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Sun Oct 4 09:44:36 2026 -0500

    [improve][test] Add a catch-up scenario: consumers joining a live stream later (#26826)
---
 tests/performance/AGENTS.md                        |   2 +-
 tests/performance/docs/run-reports.md              |  13 +-
 .../performance/launcher/ScenarioFilesTest.java    |   8 +
 .../pulsar/tests/performance/report/RunReport.java |  71 +++++++-
 .../tests/performance/report/RunReportTest.java    |  53 ++++++
 tests/performance/scenarios/README.md              |   1 +
 tests/performance/scenarios/docs/iot-telemetry.md  |  28 ++++
 .../scenarios/iot-telemetry-catch-up.yaml          |  33 ++++
 .../tests/performance/tools/CatchUpTracker.java    |  87 ++++++++++
 .../tests/performance/tools/IotScenario.java       |  44 ++++-
 .../performance/tools/MeasurementStartMarker.java  |  67 ++++++++
 .../tests/performance/tools/PerformanceTool.java   |   7 +-
 .../tests/performance/tools/TelemetryConsumer.java | 182 +++++++++++++++++++--
 .../tests/performance/tools/TelemetryProducer.java |   4 +
 .../performance/tools/CatchUpTrackerTest.java      |  64 ++++++++
 .../tests/performance/tools/IotScenarioTest.java   |  72 ++++++++
 .../tools/MeasurementStartMarkerTest.java          |  52 ++++++
 17 files changed, 760 insertions(+), 28 deletions(-)

diff --git a/tests/performance/AGENTS.md b/tests/performance/AGENTS.md
index 557a4f41ac7d..6c3b6345dfb5 100644
--- a/tests/performance/AGENTS.md
+++ b/tests/performance/AGENTS.md
@@ -307,7 +307,7 @@ commit segment can be absent). Use the exact recording names linked from the pro
 | `console.log.txt` | What the launcher printed, including the progress lines every 10 s | Read |
 | `launcher.log` | Testcontainers' and the Pulsar containers' logs; kept when the run failed, or with `-Pperformance.keepLauncherLog` | `rg` |
 | `gateways/gateways-summary.json` | The gateways' counts and throughput; `measurementMessages`, `measurementStartEpochMs` and `measurementEndEpochMs` identify the measured sends | `jq` |
-| `applications/<application>/application-summary.json` | Each application's unique messages, duplicates, ordering violations and invalid messages; `lastMeasurementMessageReceivedEpochMs` records the last measured receipt | `jq` |
+| `applications/<application>/application-summary.json` | Each application's unique messages, duplicates, ordering violations and invalid messages; `lastMeasurementMessageReceivedEpochMs` records the last measured receipt; `joinSeconds`, `joinEpochMs`, `caughtUpEpochMs`, `caughtUpLatencyMillis` and `messagesWhenCaughtUp` describe an application that joined after the measurement started (`applications.joinSeconds`), with 0 times otherwise | `jq` |
 | `applications/<application>/ordering-violations.txt` | Samples of the ordering violations; empty in a valid run | Read |
 | `gateways/container.log.txt`, `applications/container.log.txt` | The workload containers' logs | Read |
 | `gateways/gateways-latency.hgrm`, `applications/<application>/application-latency.hgrm` | The publish and end-to-end latency percentile distributions, in milliseconds, as text | Read |
diff --git a/tests/performance/docs/run-reports.md b/tests/performance/docs/run-reports.md
index 59a5a090d093..b4bd88e599e5 100644
--- a/tests/performance/docs/run-reports.md
+++ b/tests/performance/docs/run-reports.md
@@ -118,7 +118,7 @@ didn't receive every message, and the launcher fails a run when an application's
 ├── heap-dumps/                        the heap dumps, when the scenario asks for them, see heap-dumps.md
 ├── metrics.json                       the run's metrics in VictoriaMetrics and Grafana, see metrics.md
 ├── grafana-panels/                    panels of Grafana's dashboards over the run, as PNG images
-└── coordination/                      the warmup barrier markers of the gateways and the applications
+└── coordination/                      the warmup barrier and measurement start markers of the workloads
 ```
 
 A profiled run adds the recordings, their flame graphs and a profile report to each profiled component's directory,
@@ -143,6 +143,13 @@ report has these sections:
   runs can be compared.
 - **Throughput**: the gateways' throughput, the delivered throughput until the slowest application received the last
   message, the measurement's duration and how long the applications were still receiving after the gateways finished.
+- **Catch-up**, when applications join after the measurement starts (`applications.joinSeconds`): each late
+  application's join time, its subscription's backlog then, how long it took to catch up, its catch-up rate and when it
+  received its last message. An application that didn't catch up shows its rate until its last message instead. When
+  applications join later, the Throughput section's delivered throughput is mostly the last one's. The backlog chart
+  shows each subscription's backlog building up until its application joins and falling as it catches up. A late
+  application's end-to-end latency includes the time its messages waited in the backlog, so its lines dominate the
+  Latency section's charts.
 - **Latency**: the publish latency (send to acknowledgment) and each application's end-to-end latency (publish to
   consume) at percentiles from p50 to the maximum, with charts by percentile and over time.
 - **Backlog and rates**: each subscription's backlog and the per-second rates, sampled from the broker's topic
@@ -191,12 +198,12 @@ beside it, rendered with [commonmark-java](https://github.com/commonmark/commonm
 | `gateways/gateways-summary.json` | The gateways' counts and throughput, and the epoch-millisecond boundaries of the measurement |
 | `gateways/gateways-latency.hdr`, `.hgrm` | The publish latency log, and its percentile distribution in milliseconds, see [Latency logs](#latency-logs) |
 | `gateways/gateways-state.bin` | The gateways' next sequence number for each device. The launcher compares it with each application's `application-state.bin` and fails the run when they differ, which catches messages missing at the end, where no gap shows |
-| `applications/<application>/application-summary.json` | The application's unique messages, duplicates, ordering violations and invalid messages, and its first and last measured-message receipt |
+| `applications/<application>/application-summary.json` | The application's unique messages, duplicates, ordering violations and invalid messages, and its first and last measured-message receipt. Its join setting (`joinSeconds`), and for an application that joined after the measurement started, its join time (`joinEpochMs`), when it caught up (`caughtUpEpochMs`) within `caughtUpLatencyMillis`, and its received messages then (`messagesWhenCaughtUp`); `joinEpochMs` is 0 for an application  [...]
 | `applications/<application>/application-latency.hdr`, `.hgrm` | The application's end-to-end latency log, and its percentile distribution in milliseconds |
 | `applications/<application>/application-state.bin` | The application's next expected sequence number for each device |
 | `applications/<application>/ordering-violations.txt` | Samples of the ordering violations, with the message ID, topic and receiving thread; empty in a valid run |
 | `gateways/container.log.txt`, `applications/container.log.txt` | The container's log, named `.txt` so that HTTP servers show it as text |
-| `coordination/` | The markers with which the applications tell the gateways that they received a warmup round |
+| `coordination/` | The markers with which the applications tell the gateways that they received a warmup round, and the gateways' measurement start, from which the applications that join later count their join time |
 
 ## Latency logs
 
diff --git a/tests/performance/launcher/src/test/java/org/apache/pulsar/tests/performance/launcher/ScenarioFilesTest.java b/tests/performance/launcher/src/test/java/org/apache/pulsar/tests/performance/launcher/ScenarioFilesTest.java
index f45573e260c9..6163caadd3bf 100644
--- a/tests/performance/launcher/src/test/java/org/apache/pulsar/tests/performance/launcher/ScenarioFilesTest.java
+++ b/tests/performance/launcher/src/test/java/org/apache/pulsar/tests/performance/launcher/ScenarioFilesTest.java
@@ -102,6 +102,14 @@ public class ScenarioFilesTest {
         return value != null ? Integer.parseInt(value) : 2;
     }
 
+    @Test
+    public void theCatchUpScenarioJoinsApplicationsAfterTheMeasurementStarts() {
+        ObjectNode applications = (ObjectNode) resolve(SCENARIOS.resolve("iot-telemetry-catch-up.yaml"), List.of())
+                .path("workloads").path("iotTelemetry").path("applications");
+        assertThat(applications.path("joinSeconds").toString()).isEqualTo("[0,20,22,24,60]");
+        assertThat(applications.path("count").asInt()).isEqualTo(5);
+    }
+
     @Test
     public void theMemoryConfigurationsSetTheirOwnMemory() {
         // The configuration that a scenario extends last sets the memory, over the default of iot-telemetry-base.yaml
diff --git a/tests/performance/report-tool/src/main/java/org/apache/pulsar/tests/performance/report/RunReport.java b/tests/performance/report-tool/src/main/java/org/apache/pulsar/tests/performance/report/RunReport.java
index 7d611339e98c..012c8db68b7f 100644
--- a/tests/performance/report-tool/src/main/java/org/apache/pulsar/tests/performance/report/RunReport.java
+++ b/tests/performance/report-tool/src/main/java/org/apache/pulsar/tests/performance/report/RunReport.java
@@ -322,10 +322,12 @@ public final class RunReport {
         appendProfiles(report, runDirectory, mapper);
         appendCorrectness(report, run.workload(), consumers);
         appendThroughput(report, producer, consumers);
-        appendLatency(report, runDirectory, run, consumerHistograms, measurementStart);
         Path stats = runDirectory.resolve(TOPIC_STATS_FILE);
-        if (Files.isRegularFile(stats)) {
-            appendTopicStats(report, runDirectory, readSamples(stats), measurementStart, measurementEnd,
+        Samples samples = Files.isRegularFile(stats) ? readSamples(stats) : null;
+        appendCatchUp(report, run.workload(), consumers, samples, measurementStart, measurementEnd);
+        appendLatency(report, runDirectory, run, consumerHistograms, measurementStart);
+        if (samples != null) {
+            appendTopicStats(report, runDirectory, samples, measurementStart, measurementEnd,
                     run.cooldowns(), chartFooter(run.info(), run.finished()));
         }
         if (hostSamples != null) {
@@ -640,6 +642,69 @@ public final class RunReport {
                         producer.path("measurementElapsedSeconds").asDouble())))
                 .append(row("Applications still receiving after the gateways finished",
                         String.format(Locale.ROOT, "%.1f s", Math.max(0, lastReceived - end) / 1000.0)));
+        if (consumers.stream().anyMatch(consumer -> consumer.path("joinEpochMs").asLong() > 0)) {
+            report.append("\nSome applications joined after the measurement started, so the delivered throughput and"
+                    + " the time the applications were still receiving are mostly those of the last one to finish;"
+                    + " the Catch-up section has each one's.\n");
+        }
+    }
+
+    /**
+     * The applications that joined after the measurement started ({@code applications.joinSeconds}): when each
+     * joined, its subscription's backlog then, from the sampled topic stats, how long it took to catch up, and its
+     * catch-up rate.
+     */
+    static void appendCatchUp(StringBuilder report, JsonNode workload, List<JsonNode> consumers, Samples samples,
+                              long measurementStart, long measurementEnd) {
+        List<JsonNode> late = consumers.stream().filter(consumer -> consumer.path("joinEpochMs").asLong() > 0)
+                .toList();
+        if (late.isEmpty()) {
+            return;
+        }
+        int caughtUpLatency = late.get(0).path("caughtUpLatencyMillis")
+                .asInt(workload.path("applications").path("caughtUpLatencyMillis").asInt(1000));
+        report.append("\n## Catch-up\n\nThe applications that joined after the measurement started. Each had its"
+                        + " subscription from the start, so its backlog built up until it joined. An application has"
+                        + " caught up when each of its topics delivered a measured message within ")
+                .append(String.format(Locale.ROOT, "%,d", caughtUpLatency))
+                .append(" ms of its publishing; the time includes opening its pods. Its catch-up rate is the messages"
+                        + " that it received until then, warmup included, per second since it joined; for an"
+                        + " application that didn't catch up, it is the rate until its last message. The backlog is"
+                        + " the sampled topic stats' last sample before the application joined.\n\n"
+                        + "| Application | Joined | Backlog when it joined | Caught up after | Catch-up rate"
+                        + " | Received its last message after |\n|---|---:|---:|---:|---:|---:|\n");
+        for (JsonNode consumer : late) {
+            String application = applicationName(workload, consumer.path("applicationIndex").asInt());
+            long joined = consumer.path("joinEpochMs").asLong();
+            long caughtUp = consumer.path("caughtUpEpochMs").asLong();
+            long lastReceived = consumer.path("lastMeasurementMessageReceivedEpochMs").asLong();
+            String backlog = "–";
+            double[] backlogs = samples != null ? samples.backlog().get(application) : null;
+            if (backlogs != null) {
+                for (int round = samples.epochMillis().length - 1; round >= 0; round--) {
+                    if (samples.epochMillis()[round] <= joined && !Double.isNaN(backlogs[round])) {
+                        backlog = String.format(Locale.ROOT, "%,.0f", backlogs[round]);
+                        break;
+                    }
+                }
+            }
+            String caughtUpAfter;
+            String rate;
+            if (caughtUp > 0 && caughtUp >= joined) {
+                caughtUpAfter = String.format(Locale.ROOT, "%.1f s", (caughtUp - joined) / 1000.0)
+                        + (caughtUp > measurementEnd ? ", after the gateways finished" : "");
+                rate = caughtUp > joined ? String.format(Locale.ROOT, "%,.0f msg/s",
+                        consumer.path("messagesWhenCaughtUp").asLong() * 1000.0 / (caughtUp - joined)) : "–";
+            } else {
+                caughtUpAfter = "not caught up";
+                rate = lastReceived > joined ? String.format(Locale.ROOT, "%,.0f msg/s until its last message",
+                        consumer.path("uniqueMessages").asLong() * 1000.0 / (lastReceived - joined)) : "–";
+            }
+            report.append(String.format(Locale.ROOT, "| `%s` | %.1f s | %s | %s | %s | %s |%n", application,
+                    (joined - measurementStart) / 1000.0, backlog, caughtUpAfter, rate,
+                    lastReceived > joined ? String.format(Locale.ROOT, "%.1f s", (lastReceived - joined) / 1000.0)
+                            : "–"));
+        }
     }
 
     /**
diff --git a/tests/performance/report-tool/src/test/java/org/apache/pulsar/tests/performance/report/RunReportTest.java b/tests/performance/report-tool/src/test/java/org/apache/pulsar/tests/performance/report/RunReportTest.java
index 4705fd4f3e67..53ea38d01bc2 100644
--- a/tests/performance/report-tool/src/test/java/org/apache/pulsar/tests/performance/report/RunReportTest.java
+++ b/tests/performance/report-tool/src/test/java/org/apache/pulsar/tests/performance/report/RunReportTest.java
@@ -28,6 +28,7 @@ import java.nio.file.Path;
 import java.time.ZonedDateTime;
 import java.util.Comparator;
 import java.util.List;
+import java.util.Map;
 import java.util.stream.Stream;
 import org.HdrHistogram.Histogram;
 import org.HdrHistogram.HistogramLogWriter;
@@ -62,6 +63,58 @@ public class RunReportTest {
         }
     }
 
+    @Test
+    public void reportsTheCatchUpOfTheApplicationsThatJoinedLater() throws Exception {
+        JsonNode workload = mapper.readTree("{\"applications\": {\"subscriptionPrefix\": \"app-\"}}");
+        long start = 1_000_000;
+        long end = start + 120_000;
+        List<JsonNode> consumers = List.of(
+                mapper.readTree("{\"applicationIndex\": 0, \"joinEpochMs\": 0}"),
+                // caught up while the gateways published
+                late(1, start + 20_000, start + 30_000, 1_500_000, start + 120_000),
+                // caught up with the gateways' last messages, within the threshold after they finished
+                late(2, start + 40_000, start + 120_400, 4_000_000, start + 140_000),
+                // never caught up, with no sample before it joined
+                late(3, start + 500, 0, 4_600_000, start + 150_000),
+                // joined but never received a measured message
+                late(4, start + 60_000, 0, 0, 0),
+                // caught up in the millisecond that it joined
+                late(5, start + 70_000, start + 70_000, 10, start + 80_000));
+        long[] epochs = {start + 1_000, start + 19_000, start + 39_000};
+        RunReport.Samples samples = new RunReport.Samples(epochs, new double[3], Map.of(),
+                Map.of("app-1", new double[] {0, 570_000, 0}, "app-2", new double[] {0, 0, 1_170_000},
+                        "app-3", new double[] {0, 0, 0}));
+        StringBuilder report = new StringBuilder();
+        RunReport.appendCatchUp(report, workload, consumers, samples, start, end);
+
+        assertThat(report.toString())
+                .contains("## Catch-up")
+                // the threshold that the application wrote
+                .contains("within 500 ms of its publishing")
+                .contains("| `app-1` | 20.0 s | 570,000 | 10.0 s | 150,000 msg/s | 100.0 s |")
+                .contains("| `app-2` | 40.0 s | 1,170,000 | 80.4 s, after the gateways finished | 49,751 msg/s"
+                        + " | 100.0 s |")
+                .contains("| `app-3` | 0.5 s | – | not caught up | 30,769 msg/s until its last message | 149.5 s |")
+                .contains("| `app-4` | 60.0 s | – | not caught up | – | – |")
+                .contains("| `app-5` | 70.0 s | – | 0.0 s | – | 10.0 s |")
+                .doesNotContain("`app-0`");
+        // without topic stats
+        StringBuilder withoutSamples = new StringBuilder();
+        RunReport.appendCatchUp(withoutSamples, workload, consumers.subList(0, 2), null, start, end);
+        assertThat(withoutSamples.toString()).contains("| `app-1` | 20.0 s | – | 10.0 s |");
+        // without late applications, there's no section
+        StringBuilder none = new StringBuilder();
+        RunReport.appendCatchUp(none, workload, consumers.subList(0, 1), samples, start, end);
+        assertThat(none.toString()).isEmpty();
+    }
+
+    private JsonNode late(int index, long joined, long caughtUp, long messages, long lastReceived) throws Exception {
+        return mapper.readTree("{\"applicationIndex\": " + index + ", \"joinEpochMs\": " + joined
+                + ", \"caughtUpLatencyMillis\": 500, \"caughtUpEpochMs\": " + caughtUp
+                + ", \"messagesWhenCaughtUp\": " + messages + ", \"uniqueMessages\": 4600000"
+                + ", \"lastMeasurementMessageReceivedEpochMs\": " + lastReceived + "}");
+    }
+
     @Test
     public void reportsCorrectnessThroughputLatencyAndSampledStats() throws IOException {
         Files.createDirectories(run.resolve("gateways"));
diff --git a/tests/performance/scenarios/README.md b/tests/performance/scenarios/README.md
index 35d0fbc32a35..ab9eaf6e69c0 100644
--- a/tests/performance/scenarios/README.md
+++ b/tests/performance/scenarios/README.md
@@ -47,6 +47,7 @@ detail.
 | [`iot-telemetry-small-restarts.yaml`](iot-telemetry-small-restarts.yaml) | The smaller topology, restarting 10 % of each application's clients every 30 s | low, about 3 GB |
 | [`iot-telemetry-restarts.yaml`](iot-telemetry-restarts.yaml) | The full topology, restarting 10 % of each application's clients every 30 s | medium, about 11 GB |
 | [`iot-telemetry-high-rate.yaml`](iot-telemetry-high-rate.yaml) | A high rate: five million messages at 30,000 messages per second, from 500 preconnected producers to one topic, consumed by 5 applications with 10 pods each | high, about 14 GB |
+| [`iot-telemetry-catch-up.yaml`](iot-telemetry-catch-up.yaml) | Consumers joining a live stream: the high-rate scenario's 30,000 messages per second for 180 s, with 4 of its 5 applications joining 20, 22, 24 and 60 s after the measurement starts and reading their backlogs | high, about 14 GB |
 | [`iot-telemetry-max-rate.yaml`](iot-telemetry-max-rate.yaml) | The maximum rate: the 500 gateways of the high-rate scenario publish four million unbatched messages to one topic without a rate limit, consumed by one application with 20 pods, on single-copy ledgers | high, about 14 GB |
 
 ## Configurations
diff --git a/tests/performance/scenarios/docs/iot-telemetry.md b/tests/performance/scenarios/docs/iot-telemetry.md
index 376225662a7d..24a77bcdc50d 100644
--- a/tests/performance/scenarios/docs/iot-telemetry.md
+++ b/tests/performance/scenarios/docs/iot-telemetry.md
@@ -43,6 +43,22 @@ application-visible order across Key_Shared hash-range reassignment.
   workstation whose CPU runs at a fixed base frequency: without a rate limit, the gateways publish faster than the
   applications receive, and an application that falls behind can stall for tens of seconds. It uses the high-memory
   configuration.
+- [`iot-telemetry-catch-up.yaml`](../iot-telemetry-catch-up.yaml) measures how quickly consumers that join a live
+  stream catch up. The gateways publish at the high-rate scenario's 30,000 messages per second for 180 s, and 4 of
+  the 5 applications join after the measurement starts, 3 of them 2 s apart, at 20, 22 and 24 s, and the last one at
+  60 s. Each late application has its subscription from the start, so its backlog builds up until it joins, and it
+  then reads the backlog while the gateways keep publishing. The report's Catch-up section shows each application's
+  backlog when it joined and how long it took to catch up. An application that is still behind when the gateways
+  finish can't catch up, since no newer message comes; the report then shows when it received its last message.
+  - **The broker's entry cache:** a subscription without consumers isn't an active cursor, so nothing keeps the
+    entries for a late application before it joins. Once the 3 that join 2 s apart are reading, the entries that the
+    first of them reads from storage are expected to be read by the others behind it, so they stay in the cache for
+    its time to live, `managedLedgerCacheEvictionTimeThresholdMillis` (1 s), extended up to
+    `managedLedgerCacheEvictionExtendTTLOfEntriesWithRemainingExpectedReadsMaxTimes` (5) times, about 6 s, which the
+    2 s spacing is within. The others then read them from the cache. The last application is too far behind for that,
+    and reads its backlog from storage.
+  - **Where reads come from:** the managed ledger's cache hit and miss rates (`pulsar_ml_cache_hits_rate`,
+    `pulsar_ml_cache_misses_rate`) in the metrics show which reads the cache served.
 - [`iot-telemetry-max-rate.yaml`](../iot-telemetry-max-rate.yaml) runs the high-rate scenario's 500 gateways at the
   maximum rate: `rate: 0` removes the rate limit, so the gateways publish four million unbatched 128-byte messages
   as fast as the cluster takes them, with at most 100,000 in flight. One application consumes them with twenty pods
@@ -162,12 +178,24 @@ workloads:
         listenerThreads: 16
       env:                     # the applications' container, which runs every application; from the memory configuration
         PULSAR_MEM: -Xms1536m -Xmx1536m -XX:MaxDirectMemorySize=256m -XX:+UseTransparentHugePages -XX:+AlwaysPreTouch
+      joinSeconds: []          # not set in the base file: each application's join after the measurement starts, in s
+      caughtUpLatencyMillis: 1000  # not set in the base file: a late application has caught up within this latency
     behaviors:
       podRestarts:             # each application restarts this fraction of its pods every intervalSeconds
         intervalSeconds: 0
         fraction: 0.0
 ```
 
+`applications.joinSeconds` has each application join after the measurement starts: a list with a value for each
+application, in seconds, where 0 joins at the start, as every application does without the setting. An application
+that joins later creates its subscription at the earliest position when the applications start, before the gateways
+publish, so that the subscription keeps the backlog until the application opens its pods; it doesn't take part in
+the warmup. It has caught up when each of its topics has delivered a measured message within
+`applications.caughtUpLatencyMillis` of its publishing; the time includes opening its pods, and with Key_Shared
+redeliveries a few older messages may still arrive after it. The report's Catch-up section shows when each late
+application joined, its subscription's backlog then, and how long it took to catch up. The warmup and the latest join
+have to be within the timeout, which also has to leave room for the last application to catch up.
+
 `timeoutSeconds` bounds the whole workload: the applications stop waiting for messages after it and the run fails,
 the gateways give up waiting for a warmup round or the measurement's start, and the launcher waits for the containers
 a minute longer. It has to be at least the time that the warmup rounds, their delays and the measurement need at the
diff --git a/tests/performance/scenarios/iot-telemetry-catch-up.yaml b/tests/performance/scenarios/iot-telemetry-catch-up.yaml
new file mode 100644
index 000000000000..d68eeabed9d2
--- /dev/null
+++ b/tests/performance/scenarios/iot-telemetry-catch-up.yaml
@@ -0,0 +1,33 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+# Consumers joining a live stream: the gateways publish at the high-rate scenario's 30,000 msg/s, and 4 of the 5
+# applications join after the measurement starts. Each late application had its subscription from the start, so it
+# catches up on the backlog that built up until it joined, while the gateways keep publishing; the report's Catch-up
+# section shows how long each took. 3 applications join 2 s apart, at 20, 22 and 24 s, as consumers that scale up or
+# restart together do: the entries that the first of them reads from storage stay in the broker's entry cache for the
+# others, which are expected to read them within its time to live. The last one joins alone at 60 s and reads its
+# backlog from storage. The measurement is 180 s, so that the applications that join at 20 to 24 s catch up while the
+# gateways publish.
+extends: iot-telemetry-high-rate.yaml
+workloads:
+  iotTelemetry:
+    measurement:
+      messages: 5400000
+    applications:
+      joinSeconds: [0, 20, 22, 24, 60]
diff --git a/tests/performance/tools/src/main/java/org/apache/pulsar/tests/performance/tools/CatchUpTracker.java b/tests/performance/tools/src/main/java/org/apache/pulsar/tests/performance/tools/CatchUpTracker.java
new file mode 100644
index 000000000000..acf7e86dc3fa
--- /dev/null
+++ b/tests/performance/tools/src/main/java/org/apache/pulsar/tests/performance/tools/CatchUpTracker.java
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.tests.performance.tools;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.LongSupplier;
+
+/**
+ * When an application that joined after the measurement started catches up: when each of its topics has delivered a
+ * measured message received within the threshold of its publishing, after the application joined. An application
+ * that joined at the start doesn't catch up.
+ */
+final class CatchUpTracker {
+    private final int topicCount;
+    private final long thresholdMillis;
+    // each topic's first receipt of a message within the threshold
+    private final Map<String, Long> caughtUpTopics = new ConcurrentHashMap<>();
+    private final AtomicLong joinEpochMs = new AtomicLong();
+    // when it caught up and its received messages then, published together
+    private final AtomicReference<Result> caughtUp = new AtomicReference<>();
+
+    private record Result(long epochMs, long messages) {
+    }
+
+    CatchUpTracker(int topicCount, long thresholdMillis) {
+        this.topicCount = topicCount;
+        this.thresholdMillis = thresholdMillis;
+    }
+
+    void joined(long epochMs) {
+        joinEpochMs.set(epochMs);
+    }
+
+    /**
+     * Records a received measured message.
+     *
+     * @param messages the application's received messages, read when it has caught up
+     */
+    void received(String topic, long publishEpochMs, long receivedEpochMs, LongSupplier messages) {
+        if (joinEpochMs.get() == 0 || caughtUp.get() != null
+                || receivedEpochMs - publishEpochMs > thresholdMillis) {
+            return;
+        }
+        if (caughtUpTopics.putIfAbsent(topic, receivedEpochMs) == null && caughtUpTopics.size() >= topicCount) {
+            // the last topic's receipt, which another listener may have recorded after this one
+            caughtUp.compareAndSet(null, new Result(Collections.max(caughtUpTopics.values()), messages.getAsLong()));
+        }
+    }
+
+    long joinEpochMs() {
+        return joinEpochMs.get();
+    }
+
+    long caughtUpEpochMs() {
+        Result result = caughtUp.get();
+        return result != null ? result.epochMs() : 0;
+    }
+
+    long messagesWhenCaughtUp() {
+        Result result = caughtUp.get();
+        return result != null ? result.messages() : 0;
+    }
+
+    long thresholdMillis() {
+        return thresholdMillis;
+    }
+}
diff --git a/tests/performance/tools/src/main/java/org/apache/pulsar/tests/performance/tools/IotScenario.java b/tests/performance/tools/src/main/java/org/apache/pulsar/tests/performance/tools/IotScenario.java
index b3e263f9daef..7285e8e34648 100644
--- a/tests/performance/tools/src/main/java/org/apache/pulsar/tests/performance/tools/IotScenario.java
+++ b/tests/performance/tools/src/main/java/org/apache/pulsar/tests/performance/tools/IotScenario.java
@@ -19,6 +19,7 @@
 package org.apache.pulsar.tests.performance.tools;
 
 import java.util.ArrayList;
+import java.util.Collections;
 import java.util.List;
 import java.util.Locale;
 import java.util.Map;
@@ -94,9 +95,29 @@ public record IotScenario(String serviceUrl, Warmup warmup, Measurement measurem
      * @param client the client resources that every pod's client shares
      * @param env the environment variables of the applications' container, which the launcher sets, such as
      *            {@code PULSAR_MEM}
+     * @param joinSeconds when each application opens its pods, in seconds after the measurement starts, 0 for at the
+     *                    start; empty for every application at the start. An application that joins later has its
+     *                    subscription from the start, so it catches up on the backlog that builds up until it joins
+     * @param caughtUpLatencyMillis an application that joined later has caught up when it first receives a measured
+     *                              message within this time of its publishing
      */
     public record Applications(int count, int podsPerApplication, String subscriptionPrefix, Client client,
-                               Map<String, String> env) {
+                               Map<String, String> env, List<Integer> joinSeconds, Integer caughtUpLatencyMillis) {
+        /** The default of {@code caughtUpLatencyMillis}. */
+        public static final int DEFAULT_CAUGHT_UP_LATENCY_MILLIS = 1000;
+
+        public Applications {
+            // kept with any nulls, which the scenario's validation rejects with its message
+            joinSeconds = joinSeconds != null ? Collections.unmodifiableList(new ArrayList<>(joinSeconds)) : List.of();
+            if (caughtUpLatencyMillis == null) {
+                caughtUpLatencyMillis = DEFAULT_CAUGHT_UP_LATENCY_MILLIS;
+            }
+        }
+
+        public Applications(int count, int podsPerApplication, String subscriptionPrefix, Client client,
+                            Map<String, String> env) {
+            this(count, podsPerApplication, subscriptionPrefix, client, env, null, null);
+        }
     }
 
     /** The applications' client resources: the event loop's and the listener thread pool's threads. */
@@ -176,8 +197,17 @@ public record IotScenario(String serviceUrl, Warmup warmup, Measurement measurem
                         && client.listenerThreads() >= 1,
                 "The gateways' and the applications' ioThreads and listenerThreads must be at least 1");
         require(producer.maxOutstanding() >= 1, "gateways.producer.maxOutstanding must be at least 1");
+        require(applications.joinSeconds().isEmpty() || applications.joinSeconds().size() == applications.count(),
+                "applications.joinSeconds must have a value for each of the " + applications.count()
+                        + " applications, or none");
+        // the joins count from the measurement's start, after the warmup
+        long warmupTotalSeconds = minimumRuntimeSeconds - durationSeconds;
+        require(applications.joinSeconds().stream().allMatch(seconds -> seconds != null && seconds >= 0
+                        && warmupTotalSeconds + seconds < timeoutSeconds),
+                "applications.joinSeconds must be at least 0 with no null value, and the warmup ("
+                        + warmupTotalSeconds + " s) and the latest join must be within timeoutSeconds");
+        require(applications.caughtUpLatencyMillis() >= 1, "applications.caughtUpLatencyMillis must be at least 1");
         if (timeoutSeconds < minimumRuntimeSeconds) {
-            long warmupTotalSeconds = minimumRuntimeSeconds - durationSeconds;
             throw new IllegalArgumentException(String.format(Locale.ROOT, "Invalid IoT scenario: timeoutSeconds is %d,"
                             + " but the workload needs %d s: %d s of warmup (%d round(s) of %d s%s) and %d s of"
                             + " measurement%s. The applications stop waiting at the timeout, so set timeoutSeconds to"
@@ -239,6 +269,16 @@ public record IotScenario(String serviceUrl, Warmup warmup, Measurement measurem
         return applications.podsPerApplication();
     }
 
+    /** When the application opens its pods, in seconds after the measurement starts; 0 for at the start. */
+    public int joinSeconds(int applicationIndex) {
+        return applications.joinSeconds().isEmpty() ? 0 : applications.joinSeconds().get(applicationIndex);
+    }
+
+    /** Whether any application joins after the measurement starts. */
+    public boolean hasLateApplications() {
+        return applications.joinSeconds().stream().anyMatch(seconds -> seconds > 0);
+    }
+
     public long messageCount() {
         return Math.addExact(warmupMessageCount(), measurementMessageCount());
     }
diff --git a/tests/performance/tools/src/main/java/org/apache/pulsar/tests/performance/tools/MeasurementStartMarker.java b/tests/performance/tools/src/main/java/org/apache/pulsar/tests/performance/tools/MeasurementStartMarker.java
new file mode 100644
index 000000000000..e98139c50c1e
--- /dev/null
+++ b/tests/performance/tools/src/main/java/org/apache/pulsar/tests/performance/tools/MeasurementStartMarker.java
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.tests.performance.tools;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.Path;
+import java.nio.file.StandardCopyOption;
+import java.util.concurrent.TimeoutException;
+
+/**
+ * The gateways' measurement start, for the applications that join after it: a file in the coordination directory
+ * with the start's epoch milliseconds, written atomically.
+ */
+final class MeasurementStartMarker {
+    private static final long POLL_INTERVAL_MILLIS = 20;
+
+    private MeasurementStartMarker() {
+    }
+
+    static void mark(Path directory, String runId, long epochMs) throws IOException {
+        Files.createDirectories(directory);
+        Path marker = marker(directory, runId);
+        Path temporary = marker.resolveSibling(marker.getFileName() + ".tmp");
+        Files.writeString(temporary, Long.toString(epochMs), StandardCharsets.UTF_8);
+        Files.move(temporary, marker, StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING);
+    }
+
+    /**
+     * Waits for the measurement to start, and returns its epoch milliseconds.
+     *
+     * @throws TimeoutException when the measurement doesn't start before the deadline
+     */
+    static long await(Path directory, String runId, long deadlineNanos) throws Exception {
+        Path marker = marker(directory, runId);
+        while (deadlineNanos - System.nanoTime() > 0) {
+            try {
+                return Long.parseLong(Files.readString(marker, StandardCharsets.UTF_8).trim());
+            } catch (NoSuchFileException e) {
+                Thread.sleep(POLL_INTERVAL_MILLIS);
+            }
+        }
+        throw new TimeoutException("Timed out waiting for the measurement to start");
+    }
+
+    private static Path marker(Path directory, String runId) {
+        return directory.resolve("measurement-start-" + runId);
+    }
+}
diff --git a/tests/performance/tools/src/main/java/org/apache/pulsar/tests/performance/tools/PerformanceTool.java b/tests/performance/tools/src/main/java/org/apache/pulsar/tests/performance/tools/PerformanceTool.java
index 3161ef28a7a1..0de34684f157 100644
--- a/tests/performance/tools/src/main/java/org/apache/pulsar/tests/performance/tools/PerformanceTool.java
+++ b/tests/performance/tools/src/main/java/org/apache/pulsar/tests/performance/tools/PerformanceTool.java
@@ -72,7 +72,8 @@ public class PerformanceTool implements Callable<Integer> {
         Integer controlPort;
 
         @Option(names = "--run-id",
-                description = "Shared correlation ID, required when warmup is enabled; use a new ID per run")
+                description = "Shared correlation ID, required when warmup is enabled or applications join"
+                        + " later; use a new ID per run")
         String runId;
 
         IotScenario scenario() throws Exception {
@@ -81,6 +82,10 @@ public class PerformanceTool implements Callable<Integer> {
                 throw new IllegalArgumentException(
                         "Warmup requires the same --run-id for the producer and all consumers");
             }
+            if (scenario.hasLateApplications() && (runId == null || runId.isBlank())) {
+                throw new IllegalArgumentException(
+                        "Applications that join later require the same --run-id for the producer and all consumers");
+            }
             return scenario;
         }
 
diff --git a/tests/performance/tools/src/main/java/org/apache/pulsar/tests/performance/tools/TelemetryConsumer.java b/tests/performance/tools/src/main/java/org/apache/pulsar/tests/performance/tools/TelemetryConsumer.java
index 38103fd2e756..c6e0d613a85c 100644
--- a/tests/performance/tools/src/main/java/org/apache/pulsar/tests/performance/tools/TelemetryConsumer.java
+++ b/tests/performance/tools/src/main/java/org/apache/pulsar/tests/performance/tools/TelemetryConsumer.java
@@ -37,7 +37,9 @@ import java.util.concurrent.Future;
 import java.util.concurrent.ScheduledExecutorService;
 import java.util.concurrent.ThreadLocalRandom;
 import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
 import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
 import java.util.concurrent.atomic.AtomicLong;
 import java.util.concurrent.atomic.AtomicReference;
 import org.apache.pulsar.client.api.Consumer;
@@ -56,6 +58,11 @@ import picocli.CommandLine.Command;
  *
  * <p>Each application checks its own delivery and ordering, and writes its outputs into a directory named after its
  * subscription in {@code --output}, as the run report names it. The progress stream sums the applications' counts.
+ *
+ * <p>An application with {@code applications.joinSeconds} after the measurement's start creates its subscription at
+ * startup, so that the backlog builds up from the start, and opens its pods when it joins. It records when it joined
+ * and when it caught up: when it first received a measured message within {@code applications.caughtUpLatencyMillis}
+ * of its publishing.
  */
 @Command(name = "iot-consume", description = "Run the IoT applications, each consuming and validating its own "
         + "subscription")
@@ -89,11 +96,24 @@ final class TelemetryConsumer extends PerformanceTool.ScenarioCommand {
             sharedResources = SharedClientResources.create(scenario.applications().client().ioThreads(),
                     scenario.applications().client().listenerThreads());
             openPods(applications, sharedResources);
+            for (Application application : applications) {
+                if (application.joinsLater()) {
+                    // it doesn't take part in the warmup, which the gateways would otherwise wait for
+                    application.skipWarmup(coordinationDirectory(), runId);
+                }
+            }
             phase.set("receiving");
+            // The clients open now; the late applications' pods open when they join
             System.out.println("READY applications=" + applications.size() + " clients="
-                    + (long) applications.size() * scenario.podsPerApplication());
+                    + applications.stream().filter(application -> !application.joinsLater()).count()
+                    * scenario.podsPerApplication());
+            long deadlineNanos = System.nanoTime() + Duration.ofSeconds(scenario.timeoutSeconds()).toNanos();
             for (Application application : applications) {
-                application.startRestarts(sharedResources);
+                if (application.joinsLater()) {
+                    application.scheduleJoin(sharedResources, coordinationDirectory(), runId, deadlineNanos);
+                } else {
+                    application.startRestarts(sharedResources);
+                }
             }
             boolean succeeded = receive(scenario, applications);
             phase.set("finished");
@@ -129,6 +149,7 @@ final class TelemetryConsumer extends PerformanceTool.ScenarioCommand {
             for (Iterator<Application> iterator = receiving.iterator(); iterator.hasNext(); ) {
                 Application application = iterator.next();
                 application.checkRestarts();
+                application.checkJoin();
                 application.markWarmupRounds(coordinationDirectory(), runId);
                 if (application.receivedEveryMessage()) {
                     succeeded &= application.finish();
@@ -142,6 +163,7 @@ final class TelemetryConsumer extends PerformanceTool.ScenarioCommand {
         // The applications that timed out write what they received too
         for (Application application : receiving) {
             application.checkRestarts();
+            application.checkJoin();
             succeeded &= application.finish();
         }
         return succeeded;
@@ -154,7 +176,8 @@ final class TelemetryConsumer extends PerformanceTool.ScenarioCommand {
      */
     private static void openPods(List<Application> applications, PulsarClientSharedResources sharedResources)
             throws Exception {
-        long pods = (long) applications.size() * applications.get(0).scenario.podsPerApplication();
+        long pods = applications.stream().filter(application -> !application.joinsLater()).count()
+                * applications.get(0).scenario.podsPerApplication();
         AtomicLong opened = new AtomicLong();
         ExecutorService executor = Executors.newFixedThreadPool(Math.min(applications.size(), MAX_PARALLEL_STARTS),
                 runnable -> {
@@ -174,7 +197,11 @@ final class TelemetryConsumer extends PerformanceTool.ScenarioCommand {
             List<Future<?>> starts = new ArrayList<>(applications.size());
             for (Application application : applications) {
                 starts.add(executor.submit(() -> {
-                    application.openPods(sharedResources, opened);
+                    if (application.joinsLater()) {
+                        application.createSubscription(sharedResources);
+                    } else {
+                        application.openPods(sharedResources, opened);
+                    }
                     return null;
                 }));
             }
@@ -232,11 +259,16 @@ final class TelemetryConsumer extends PerformanceTool.ScenarioCommand {
         private final HdrLatencyRecorder receiveLatency;
         private final AtomicLong firstMeasurementReceiptEpochMs = new AtomicLong();
         private final AtomicLong lastMeasurementReceiptEpochMs = new AtomicLong();
+        // For an application that joins later: when it joined and when it caught up
+        private final CatchUpTracker catchUp;
+        private final AtomicReference<Throwable> joinFailure = new AtomicReference<>();
+        private final AtomicInteger listenersInFlight = new AtomicInteger();
+        private volatile Thread joiner;
         // Guarded by itself, as the restarts replace pods
         private final List<ClientAndConsumer> pods;
         private final AtomicBoolean stopping = new AtomicBoolean();
         private final AtomicReference<Throwable> restartFailure = new AtomicReference<>();
-        private Thread restarter;
+        private volatile Thread restarter;
         private int nextWarmupRound = 1;
         private volatile boolean finished;
 
@@ -245,6 +277,7 @@ final class TelemetryConsumer extends PerformanceTool.ScenarioCommand {
             this.index = index;
             this.output = output;
             tracker = new DeviceSequenceTracker(scenario.deviceCount());
+            catchUp = new CatchUpTracker(scenario.topicCount(), scenario.applications().caughtUpLatencyMillis());
             pods = new ArrayList<>(scenario.podsPerApplication());
             receiveLatency = new HdrLatencyRecorder(output.resolve("application-latency.hdr"),
                     PerformanceTool.MAX_LATENCY_MICROS);
@@ -254,10 +287,83 @@ final class TelemetryConsumer extends PerformanceTool.ScenarioCommand {
             return receiveLatency;
         }
 
+        boolean joinsLater() {
+            return scenario.joinSeconds(index) > 0;
+        }
+
+        /**
+         * Creates the application's subscription on every topic at the earliest position, without keeping a
+         * consumer, so that it keeps the backlog until the application joins.
+         */
+        void createSubscription(PulsarClientSharedResources sharedResources) throws Exception {
+            try (PulsarClient client = PulsarClient.builder()
+                    .serviceUrl(scenario.serviceUrl())
+                    .sharedResources(sharedResources)
+                    .build()) {
+                client.newConsumer(Schema.BYTES)
+                        .topics(scenario.topicNames())
+                        .subscriptionName(scenario.subscriptionName(index))
+                        .subscriptionType(SubscriptionType.Key_Shared)
+                        .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
+                        .subscribe()
+                        .close();
+            }
+        }
+
+        /** Marks every warmup round received, so that the gateways don't wait for an application that joins later. */
+        void skipWarmup(Path coordinationDirectory, String runId) throws IOException {
+            for (; nextWarmupRound <= scenario.warmupRounds() && scenario.warmupMessageCountPerRound() > 0;
+                 nextWarmupRound++) {
+                WarmupBarrier.markApplicationComplete(coordinationDirectory, runId, nextWarmupRound, index);
+            }
+        }
+
+        /** Opens the application's pods at its join time, counted from the gateways' measurement start. */
+        void scheduleJoin(PulsarClientSharedResources sharedResources, Path coordinationDirectory, String runId,
+                          long deadlineNanos) {
+            joiner = new Thread(() -> {
+                try {
+                    long measurementStart = MeasurementStartMarker.await(coordinationDirectory, runId, deadlineNanos);
+                    long waitMillis = measurementStart + TimeUnit.SECONDS.toMillis(scenario.joinSeconds(index))
+                            - System.currentTimeMillis();
+                    if (waitMillis > 0) {
+                        Thread.sleep(waitMillis);
+                    }
+                    // when it starts to open its pods, so the catch-up includes connecting them
+                    long joinEpochMs = System.currentTimeMillis();
+                    catchUp.joined(joinEpochMs);
+                    System.out.println("JOIN application=" + index + " epochMs=" + joinEpochMs);
+                    openPods(sharedResources, new AtomicLong());
+                    startRestarts(sharedResources);
+                } catch (InterruptedException interrupted) {
+                    Thread.currentThread().interrupt();
+                } catch (TimeoutException measurementDidNotStart) {
+                    // the receive loop's timeout reports the run as failed and writes the applications' summaries
+                    System.out.println("JOIN application=" + index + " didn't join: the measurement didn't start");
+                } catch (Throwable error) {
+                    joinFailure.compareAndSet(null, error);
+                }
+            }, "iot-application-join-" + index);
+            joiner.setDaemon(true);
+            joiner.start();
+        }
+
+        void checkJoin() {
+            if (joinFailure.get() != null) {
+                throw new IllegalStateException("Application " + index + " couldn't join", joinFailure.get());
+            }
+        }
+
         void openPods(PulsarClientSharedResources sharedResources, AtomicLong opened) throws Exception {
             for (int pod = 0; pod < scenario.podsPerApplication(); pod++) {
                 ClientAndConsumer created = createPod(sharedResources, pod);
                 synchronized (pods) {
+                    // a late application's joiner can open a pod after the application stopped
+                    if (stopping.get()) {
+                        // uninterruptibly, so that the client is closed before the shared resources are
+                        created.closeAsync().exceptionally(failure -> null).join();
+                        return;
+                    }
                     pods.add(created);
                 }
                 opened.incrementAndGet();
@@ -265,9 +371,10 @@ final class TelemetryConsumer extends PerformanceTool.ScenarioCommand {
         }
 
         void startRestarts(PulsarClientSharedResources sharedResources) {
-            if (scenario.behaviors().podRestarts().enabled()) {
-                restarter = new Thread(() -> restartPods(sharedResources), "iot-client-restarter-" + index);
-                restarter.start();
+            if (scenario.behaviors().podRestarts().enabled() && !stopping.get()) {
+                Thread thread = new Thread(() -> restartPods(sharedResources), "iot-client-restarter-" + index);
+                restarter = thread;
+                thread.start();
             }
         }
 
@@ -296,6 +403,7 @@ final class TelemetryConsumer extends PerformanceTool.ScenarioCommand {
          */
         boolean finish() throws Exception {
             stopRestarts();
+            awaitListeners();
             DeviceSequenceTracker.Summary summary = tracker.summary();
             receiveLatency.close();
             tracker.writeState(output.resolve("application-state.bin"));
@@ -309,7 +417,12 @@ final class TelemetryConsumer extends PerformanceTool.ScenarioCommand {
                     + "  \"firstMeasurementMessageReceivedEpochMs\": "
                     + firstMeasurementReceiptEpochMs.get() + ",\n"
                     + "  \"lastMeasurementMessageReceivedEpochMs\": "
-                    + lastMeasurementReceiptEpochMs.get() + "\n}\n");
+                    + lastMeasurementReceiptEpochMs.get() + ",\n"
+                    + "  \"joinSeconds\": " + scenario.joinSeconds(index) + ",\n"
+                    + "  \"joinEpochMs\": " + catchUp.joinEpochMs() + ",\n"
+                    + "  \"caughtUpLatencyMillis\": " + catchUp.thresholdMillis() + ",\n"
+                    + "  \"caughtUpEpochMs\": " + catchUp.caughtUpEpochMs() + ",\n"
+                    + "  \"messagesWhenCaughtUp\": " + catchUp.messagesWhenCaughtUp() + "\n}\n");
             closePods();
             finished = true;
             return summary.valid() && summary.uniqueMessages() == scenario.messageCount();
@@ -318,17 +431,38 @@ final class TelemetryConsumer extends PerformanceTool.ScenarioCommand {
         /** Closes the application's pods, when it hasn't finished, such as after a failure. */
         void close() throws Exception {
             stopping.set(true);
-            if (restarter != null) {
-                restarter.interrupt();
-            }
+            // so that they don't open pods after they're closed, or use the shared resources after they are
+            interruptAndAwait(joiner);
+            interruptAndAwait(restarter);
             closePods();
         }
 
         private void stopRestarts() throws InterruptedException {
             stopping.set(true);
-            if (restarter != null) {
-                restarter.interrupt();
-                restarter.join(TimeUnit.SECONDS.toMillis(10));
+            interruptAndAwait(joiner);
+            interruptAndAwait(restarter);
+        }
+
+        /**
+         * Waits briefly for the listeners that are handling a message, such as the one that received the last message
+         * and may record the catch-up with it, so that the summary includes what they record.
+         */
+        private void awaitListeners() throws InterruptedException {
+            long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(1);
+            while (listenersInFlight.get() > 0 && deadline - System.nanoTime() > 0) {
+                Thread.sleep(1);
+            }
+        }
+
+        /** Interrupts the thread and waits for it, which takes up to a client's close timeout when it opens a pod. */
+        private static void interruptAndAwait(Thread thread) throws InterruptedException {
+            if (thread == null) {
+                return;
+            }
+            thread.interrupt();
+            thread.join(TimeUnit.SECONDS.toMillis(90));
+            if (thread.isAlive()) {
+                System.err.println("WARN " + thread.getName() + " didn't stop within 90 s");
             }
         }
 
@@ -361,6 +495,7 @@ final class TelemetryConsumer extends PerformanceTool.ScenarioCommand {
                         .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
                         .messageListener((currentConsumer, message) -> {
                             long receivedEpochMs = System.currentTimeMillis();
+                            listenersInFlight.incrementAndGet();
                             try {
                                 TelemetryMessage.Decoded decoded = TelemetryMessage.decode(message.getData());
                                 byte[] key = message.getKeyBytes();
@@ -380,16 +515,24 @@ final class TelemetryConsumer extends PerformanceTool.ScenarioCommand {
                                 tracker.received(decoded.deviceId(), decoded.sequence(), message.getMessageId(),
                                         decoded.sentNanos(), message.getTopicName(),
                                         Thread.currentThread().getName());
+                                if (decoded.measurement()) {
+                                    // after counting the message, which then counts for the catch-up
+                                    catchUp.received(message.getTopicName(), message.getPublishTime(),
+                                            receivedEpochMs, tracker::uniqueMessages);
+                                }
                                 currentConsumer.acknowledgeAsync(message);
                             } catch (RuntimeException error) {
                                 tracker.invalidMessage();
                                 currentConsumer.negativeAcknowledge(message);
+                            } finally {
+                                listenersInFlight.decrementAndGet();
                             }
                         })
                         .subscribe();
                 return new ClientAndConsumer(client, consumer);
             } catch (Throwable error) {
-                client.close();
+                // uninterruptibly: an interrupted subscription would otherwise leave the client closing
+                client.closeAsync().exceptionally(failure -> null).join();
                 throw error;
             }
         }
@@ -421,8 +564,11 @@ final class TelemetryConsumer extends PerformanceTool.ScenarioCommand {
     private record ClientAndConsumer(PulsarClient client, Consumer<byte[]> consumer) implements AutoCloseable {
         @Override
         public void close() throws Exception {
-            consumer.close();
-            client.close();
+            try {
+                consumer.close();
+            } finally {
+                client.close();
+            }
         }
 
         /** Closes the consumer, then the client, also when closing the consumer failed. */
diff --git a/tests/performance/tools/src/main/java/org/apache/pulsar/tests/performance/tools/TelemetryProducer.java b/tests/performance/tools/src/main/java/org/apache/pulsar/tests/performance/tools/TelemetryProducer.java
index a8f5966fef79..1949015313a1 100644
--- a/tests/performance/tools/src/main/java/org/apache/pulsar/tests/performance/tools/TelemetryProducer.java
+++ b/tests/performance/tools/src/main/java/org/apache/pulsar/tests/performance/tools/TelemetryProducer.java
@@ -155,6 +155,10 @@ final class TelemetryProducer extends PerformanceTool.ScenarioCommand {
                     this.measurementStartEpochMs = measurementStartEpochMs;
                     phase = "measurement";
                     System.out.println("MEASUREMENT_START epochMs=" + measurementStartEpochMs);
+                    if (scenario.hasLateApplications()) {
+                        // the applications that join later count their join time from it
+                        MeasurementStartMarker.mark(coordinationDirectory(), runId, measurementStartEpochMs);
+                    }
                 }
                 long deviceSequence = deviceSequences[device]++;
                 long producerSequence = producerSequences[producerIndex]++;
diff --git a/tests/performance/tools/src/test/java/org/apache/pulsar/tests/performance/tools/CatchUpTrackerTest.java b/tests/performance/tools/src/test/java/org/apache/pulsar/tests/performance/tools/CatchUpTrackerTest.java
new file mode 100644
index 000000000000..68dabacb601c
--- /dev/null
+++ b/tests/performance/tools/src/test/java/org/apache/pulsar/tests/performance/tools/CatchUpTrackerTest.java
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.tests.performance.tools;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import org.testng.annotations.Test;
+
+public class CatchUpTrackerTest {
+    @Test
+    public void catchesUpWhenEveryTopicDeliveredWithinTheThresholdAfterJoining() {
+        CatchUpTracker tracker = new CatchUpTracker(2, 1_000);
+        // before the application joins, nothing counts
+        tracker.received("t0", 10_000, 10_100, () -> 1);
+        tracker.joined(20_000);
+        assertThat(tracker.caughtUpEpochMs()).isZero();
+        // an old message, beyond the threshold, then one topic caught up at the threshold's boundary
+        tracker.received("t0", 1_000, 21_000, () -> 2);
+        tracker.received("t0", 21_000, 22_000, () -> 3);
+        assertThat(tracker.caughtUpEpochMs()).isZero();
+        // the other topic too: caught up, with the messages received by then
+        tracker.received("t1", 22_500, 23_000, () -> 400);
+        assertThat(tracker.caughtUpEpochMs()).isEqualTo(23_000);
+        assertThat(tracker.messagesWhenCaughtUp()).isEqualTo(400);
+        // the first catch-up counts
+        tracker.received("t1", 30_000, 30_001, () -> 500);
+        assertThat(tracker.caughtUpEpochMs()).isEqualTo(23_000);
+        assertThat(tracker.messagesWhenCaughtUp()).isEqualTo(400);
+        assertThat(tracker.joinEpochMs()).isEqualTo(20_000);
+        assertThat(tracker.thresholdMillis()).isEqualTo(1_000);
+    }
+
+    @Test
+    public void catchesUpAtTheLatestTopicsReceiptWhenListenersRecordOutOfOrder() {
+        CatchUpTracker tracker = new CatchUpTracker(2, 1_000);
+        tracker.joined(20_000);
+        // concurrent listeners: t1's receipt at 23,000 is recorded before t0's earlier one at 22,000
+        tracker.received("t1", 22_900, 23_000, () -> 10);
+        tracker.received("t0", 21_900, 22_000, () -> 11);
+        assertThat(tracker.caughtUpEpochMs()).isEqualTo(23_000);
+    }
+
+    @Test
+    public void anApplicationThatJoinedAtTheStartDoesntCatchUp() {
+        CatchUpTracker tracker = new CatchUpTracker(1, 1_000);
+        tracker.received("t0", 1_000, 1_010, () -> 1);
+        assertThat(tracker.caughtUpEpochMs()).isZero();
+    }
+}
diff --git a/tests/performance/tools/src/test/java/org/apache/pulsar/tests/performance/tools/IotScenarioTest.java b/tests/performance/tools/src/test/java/org/apache/pulsar/tests/performance/tools/IotScenarioTest.java
index 4f2ae2955cd0..a3c54820743c 100644
--- a/tests/performance/tools/src/test/java/org/apache/pulsar/tests/performance/tools/IotScenarioTest.java
+++ b/tests/performance/tools/src/test/java/org/apache/pulsar/tests/performance/tools/IotScenarioTest.java
@@ -23,6 +23,7 @@ import static org.assertj.core.api.Assertions.assertThatThrownBy;
 import java.nio.file.Files;
 import java.nio.file.Path;
 import java.util.ArrayList;
+import java.util.Arrays;
 import java.util.List;
 import java.util.Map;
 import org.apache.pulsar.tests.performance.common.YamlScenarioLoader;
@@ -97,6 +98,65 @@ public class IotScenarioTest {
         assertThat(scenario.warmupMessageCount()).isEqualTo(1_000);
     }
 
+    @Test
+    public void applicationsThatJoinLaterRequireARunId() throws Exception {
+        Path config = Files.createTempFile("iot-scenario", ".yaml");
+        try {
+            YamlScenarioLoader loader = new YamlScenarioLoader();
+            loader.mapper().writeValue(config.toFile(), Map.of("workloads", Map.of("iotTelemetry",
+                    scenario(new IotScenario.Applications(2, 2, "app-", new IotScenario.Client(2, 2), null,
+                            List.of(0, 20), null)))));
+            List<String> arguments = new ArrayList<>(List.of("iot-consume", "--config", config.toString(),
+                    "--output", "results"));
+            var parsed = new CommandLine(new PerformanceTool()).parseArgs(arguments.toArray(String[]::new));
+            var tool = (PerformanceTool.ScenarioCommand) parsed.subcommand().commandSpec().userObject();
+            assertThatThrownBy(tool::scenario).isInstanceOf(IllegalArgumentException.class)
+                    .hasMessageContaining("join later require the same --run-id");
+            arguments.addAll(List.of("--run-id", "shared-run"));
+            parsed = new CommandLine(new PerformanceTool()).parseArgs(arguments.toArray(String[]::new));
+            tool = (PerformanceTool.ScenarioCommand) parsed.subcommand().commandSpec().userObject();
+            assertThat(tool.scenario().joinSeconds(1)).isEqualTo(20);
+        } finally {
+            Files.deleteIfExists(config);
+        }
+    }
+
+    @Test
+    public void joinsApplicationsLater() {
+        IotScenario scenario = scenario(new IotScenario.Applications(3, 2, "app-", new IotScenario.Client(2, 2), null,
+                List.of(0, 20, 20), null));
+        assertThat(scenario.joinSeconds(0)).isZero();
+        assertThat(scenario.joinSeconds(1)).isEqualTo(20);
+        assertThat(scenario.hasLateApplications()).isTrue();
+        assertThat(scenario.applications().caughtUpLatencyMillis())
+                .isEqualTo(IotScenario.Applications.DEFAULT_CAUGHT_UP_LATENCY_MILLIS);
+        // without the setting, every application joins at the start
+        IotScenario atStart = scenario(new IotScenario.Applications(3, 2, "app-", new IotScenario.Client(2, 2), null));
+        assertThat(atStart.joinSeconds(2)).isZero();
+        assertThat(atStart.hasLateApplications()).isFalse();
+    }
+
+    @Test
+    public void rejectsInvalidJoinSettings() {
+        assertThatThrownBy(() -> scenario(new IotScenario.Applications(3, 2, "app-", new IotScenario.Client(2, 2),
+                null, List.of(0, 20), null))).hasMessageContaining("a value for each of the 3 applications");
+        assertThatThrownBy(() -> scenario(new IotScenario.Applications(2, 2, "app-", new IotScenario.Client(2, 2),
+                null, List.of(0, -1), null))).hasMessageContaining("must be at least 0");
+        assertThatThrownBy(() -> scenario(new IotScenario.Applications(2, 2, "app-", new IotScenario.Client(2, 2),
+                null, List.of(0, 300), null))).hasMessageContaining("the latest join must be within timeoutSeconds");
+        assertThatThrownBy(() -> scenario(new IotScenario.Applications(2, 2, "app-", new IotScenario.Client(2, 2),
+                null, Arrays.asList(0, null), null))).hasMessageContaining("applications.joinSeconds must be");
+        assertThatThrownBy(() -> scenario(new IotScenario.Applications(2, 2, "app-", new IotScenario.Client(2, 2),
+                null, List.of(0, 20), 0))).hasMessageContaining("caughtUpLatencyMillis must be at least 1");
+        // the joins count from the measurement's start, after 2 warmup rounds of 20 s with 2 s after each
+        IotScenario.Warmup warmup = new IotScenario.Warmup(20, 0, 2, 2);
+        assertThat(scenario(new IotScenario.Applications(2, 2, "app-", new IotScenario.Client(2, 2), null,
+                List.of(0, 255), null), warmup).joinSeconds(1)).isEqualTo(255);
+        assertThatThrownBy(() -> scenario(new IotScenario.Applications(2, 2, "app-", new IotScenario.Client(2, 2),
+                null, List.of(0, 256), null), warmup))
+                .hasMessageContaining("the warmup (44 s) and the latest join must be within timeoutSeconds");
+    }
+
     @Test
     public void rejectsTimeBasedWarmupWithoutRateLimit() {
         assertThatThrownBy(() -> scenario(20, 0, 0, 5_000_000))
@@ -129,6 +189,18 @@ public class IotScenarioTest {
                 300);
     }
 
+    private static IotScenario scenario(IotScenario.Applications applications) {
+        return scenario(applications, new IotScenario.Warmup(0, 0, 1, 0));
+    }
+
+    private static IotScenario scenario(IotScenario.Applications applications, IotScenario.Warmup warmup) {
+        // a time-based warmup needs a rate
+        return new IotScenario("pulsar://localhost:6650", warmup, new IotScenario.Measurement(120, 1_000),
+                warmup.seconds() > 0 ? 100 : 0, new IotScenario.Payload(64), new IotScenario.Devices(1_000),
+                new IotScenario.Gateways(10, new IotScenario.Producer(2, 2, 100, true, true), null),
+                new IotScenario.Topics(2, "persistent://public/default/iot-"), applications, null, 300);
+    }
+
     private static IotScenario scenario(int warmupSeconds, long warmupMessages, int warmupRounds,
                                         int warmupRoundDelaySeconds, int rate, long numberOfMessages,
                                         int timeoutSeconds) {
diff --git a/tests/performance/tools/src/test/java/org/apache/pulsar/tests/performance/tools/MeasurementStartMarkerTest.java b/tests/performance/tools/src/test/java/org/apache/pulsar/tests/performance/tools/MeasurementStartMarkerTest.java
new file mode 100644
index 000000000000..55a62fe8e2ea
--- /dev/null
+++ b/tests/performance/tools/src/test/java/org/apache/pulsar/tests/performance/tools/MeasurementStartMarkerTest.java
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.tests.performance.tools;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import org.testng.annotations.Test;
+
+public class MeasurementStartMarkerTest {
+    @Test
+    public void applicationsAwaitTheGatewaysMeasurementStart() throws Exception {
+        Path directory = Files.createTempDirectory("coordination");
+        CompletableFuture<Long> awaited = CompletableFuture.supplyAsync(() -> {
+            try {
+                return MeasurementStartMarker.await(directory, "run-1",
+                        System.nanoTime() + TimeUnit.SECONDS.toNanos(10));
+            } catch (Exception e) {
+                throw new IllegalStateException(e);
+            }
+        });
+        Thread.sleep(50);
+        assertThat(awaited).isNotDone();
+        MeasurementStartMarker.mark(directory, "run-1", 1_790_000_000_000L);
+        assertThat(awaited.get(10, TimeUnit.SECONDS)).isEqualTo(1_790_000_000_000L);
+        // another run's marker isn't this run's
+        assertThatThrownBy(() -> MeasurementStartMarker.await(directory, "run-2",
+                System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(100)))
+                .isInstanceOf(TimeoutException.class)
+                .hasMessageContaining("Timed out waiting for the measurement to start");
+    }
+}