You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by lh...@apache.org on 2026/09/28 07:23:43 UTC
(pulsar) branch lh-key-shared-500x20-scenario updated (beb39f3df0d2 -> 1d34213ee7b5)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a change to branch lh-key-shared-500x20-scenario
in repository https://gitbox.apache.org/repos/asf/pulsar.git
discard beb39f3df0d2 [improve][test] Keep the Key_Shared 500x20 scenario saturating after the high-rate scenario got a rate limit
discard 87140007ce0b [improve][test] Move the Key_Shared 500x20 scenarios to the restructured scenario settings
discard 791bf10cd343 [improve][test] Name the Key_Shared 500x20 runs after their scenario files
discard 8fb8f40b2182 [improve][test] Add a launcher scenario for 500 producers and a 20-member Key_Shared subscription
omit edd286585aee [improve][test] Name the same tested features for Linux and macOS in the README
omit 37e5f926d979 [improve][test] Link each Grafana panel of the run report to the dashboard that it is on
omit 1e2f81923c10 [fix][test] Address a review of the profile report's flame graph descriptions
omit f7e5ffe16ce8 [improve][test] Point agents to the jonoffcpu report and the collapsed stacks for blocked time
omit 73fcebf3bc95 [improve][test] Explain what the off-CPU and async-profiler flame graphs show, and name the blocked time as such
omit 14e96cd725f7 [improve][test] Lead each profile's links with the jonoffcpu report (off-CPU summary)
omit 940da4d2e71e [improve][test] Describe what the SSH tunnel prints, and how -v shows the forwarding
omit e626ae27408f [improve][test] Document an SSH tunnel for the reports, Grafana and VictoriaMetrics, and link vmui's views
omit 3b29d21a9f8c [improve][test] Link each profile's reports from the run report, and bring the performance tests' guides up to date
omit 4f9168008bcb [fix][test] Address a review of the performance tests' profiling, heap dumps, sampling and metrics stack
omit e06bbda0dbb4 [improve][test] Collect the performance tests' Pulsar metrics into VictoriaMetrics and show them on Grafana's Pulsar dashboards
omit 91b55e1748f4 [improve][test] Print the performance reports' URLs on the reports server, with a configurable base URL
omit fcb11cb01a7d [improve][test] Keep the performance launcher's console output in the run directory, and delete launcher.log of successful runs
omit 68ce52f92dbc [improve][test] Let the performance tests write their heap dumps gzip-compressed
omit 440a3d1b3904 [fix][test] Run HealthCheckTest with the proxy it had before the cluster honored numProxies
omit 55d21e5335e7 [improve][test] Show the full IoT topology's progress lines from a run with the current memory settings
omit b9f023b2f79d [improve][test] Write heap dumps of the performance tests' JVMs on demand, and analyze them with jafar-shell
omit fda4cf6630cc [improve][test] Give the performance tests' broker the heap its scenario needs, and a sustainable high-rate scenario
omit 7cee2b8580a1 [improve][test] Show comparing with a released Pulsar in the performance testing README
omit 19c3aabb7537 [improve][test] Let the performance tests run the cluster on a released Pulsar
omit 3375d41cf760 [improve][test] Upgrade jonoffcpu to 0.8.0
omit af6040b6378f [improve][test] Use a jonoffcpu 0.8.0 snapshot from the local Maven repository
omit 586621337cc3 [fix][test] Keep async-profiler's events in the measurement recording when its clock has another origin
omit e4cccc23d26f [fix][test] Check the IoT workload's settings before a performance run starts a cluster
omit a895f192d263 [fix][test] Let the performance launcher write into the directories that the workloads create on a Linux host
omit 52dbc020e092 [improve][test] Close the IoT applications' pods at the same time, and size every memory configuration's applications heap
omit b425a79645cf [improve][test] Fail a performance run as soon as a workload can't start, and give the applications the heap they need
omit b774970cf323 [improve][test] Run the IoT applications in one container, and set the performance scenarios' memory with configurations
omit 0427c7b54c8d [improve][test] Give the performance workloads' containers their own env, and bring the docs up to date
omit ae5170fea457 [improve][test] Structure the performance scenarios' cluster and IoT workload settings
omit ef662192dd0f [improve][test] Profile each performance test component on its own, and name the workloads gateways and applications
omit b9df1f8c4052 [improve][test] Add scenarios and settings on the performance launcher's command line, and profile with profile files
omit 920dbcb6b730 [improve][test] Recommend a Linux host in the performance tests' prerequisites, and drop the duplicated Docker item
omit 25bd07c0aa0c [improve][test] Describe which hosts can run the performance tests, and why a Linux host gives consistent results
omit a7df75b49f08 [fix][test] Give a cluster without a proxy a broker's HTTP service URL
omit c4f9d7713879 [improve][test] Accept the legacy profiling runner's overrides with a lower-case prefix
omit ef191a87e69a [improve][test] Link the performance scenario override examples to how overrides work
omit cae3867c1464 [fix][test] Pass the performance scenario's environment overrides to the launcher when Gradle reuses its configuration
omit 744f50520d91 [improve][test] Move the legacy profiling runner's scenarios and docs beside the runner
omit 156b27eaa428 [improve][test] Run the full IoT telemetry scenario in the performance testing docs, and rename the smaller ones
omit aa7559667717 [fix][test] Keep the performance run's backlog on the console current
omit a60fed394a15 [improve][test] Print the scenario's name beside its file name when a performance run starts
omit 118c5082f2b2 [improve][test] Print the scenario and its resolved configuration when a performance run starts
omit 63bf7b1c49ca [fix][test] Start the integration test cluster's proxy only when its spec asks for one
omit e6668cdf5b4b [improve][test] Fail a performance run as soon as a workload fails, with its cause in one line
omit 3fcbcfda557d [improve][test] Report the performance run's progress on the console instead of the container logs
omit 805df48739a9 [improve][test] Replace the deprecated configuration delegate in the performance launcher build
omit d40b49798071 [improve][test] Install the performance testing profile without activating it
omit af50a76b6a04 [improve][test] Install TuneD only when it is missing, and check for jq only where it's used
omit 5f2eb8c1b3f9 [fix][test] Keep the host's dirty-page limits in the performance testing profile
omit e9b9cb6c01b3 [fix][test] Don't assume the test host's CPU topology in the host details tests
omit a2f530a8d64a [improve][test] Stream the host details from JFR instead of reading back a dumped recording
omit f73994dcb5b9 [improve][test] Record the host's hardware with JFR, and the Docker engine, in a run's information
omit 3b667c9e9c5b [improve][test] Title a run report by its start, branch, commit and scenario, and link it to the guide
omit 68d9d00aae7e [improve][test] Name a performance run on a detached HEAD after the closest branch that contains its commit
omit 5ec1a96ca280 [improve][doc] Guide agents to attach the performance charts to Pulsar PRs
omit 361051836967 [improve][test] Rename the reports server's address property to bindAddress, and document reaching the server
omit a817cbd35b67 [improve][doc] Set up a shared reports root before running the performance tests
omit dd1c7d48dc64 [improve][test] Point to JDK Mission Control and its Flight Recorder guide from the profile reports and the docs
omit b830bb5c0e84 [improve][doc] Link the profilers in the performance testing guide
omit 9a9717570445 [improve][doc] Say that the performance tests aren't automated regression tests yet
omit 53c32845fc71 [improve][doc] Name the analysis tools in one place, and link to the performance testing guide from other docs
omit fd6e15569f15 [improve][doc] Describe the heap dump analysis tools in one section, and link to it
omit 972263815a11 [improve][doc] Mention the jafar-perf plugin for analyzing JFR recordings and heap dumps
omit 8644310e4eff [improve][doc] Explain that the performance tests simulate real-world use cases, with the IoT domain they model
omit 0f3a7ef291b0 [improve][doc] Introduce the performance tests in the tests README
omit 41318da8a463 [improve][test] Render the latency charts as SVG too, and let every SVG chart scale to the space it gets
omit 76c136f52b69 [improve][build] Keep libraries that only tools use out of pulsar-dependencies
omit e203f22faa69 [improve][doc] Describe a baseline that has the same performance tests, and what to do when a run fails
omit 2aa1d3e0de50 [fix][test] Wait for the recorded interval and the host sample instead of sleeping in the performance tests
omit cb7f5f94fba4 [improve][test] Collapse the host's temperature and frequency charts in the run report
omit 985af4c25c94 [fix][test] Report the producer's message counts, and chart the backlog and rates per subscription
omit 840a4386c534 [fix][test] Count thermal throttle events on every core, and report unknown throttling as unknown
omit 76f859928681 [fix][test] Write the user's Gradle properties as the user, and restart Docker only when no containers run
omit 457544f9ff85 [improve][doc] Describe the recorders of a profiled performance run consistently
omit b2a866a97919 [improve][test] Add a validate command that checks the host before performance tests
omit bebaabf3939f [improve][doc] Summarize the test areas in CONTRIBUTING.md and add agent guides for them
omit 00ff715a692e [improve][doc] Restructure the performance testing docs as a tutorial with reference pages
omit f9db3d128eda [improve][test] Name the run and profile reports README.md and index.html
omit b58f0592fe8b [improve][test] Add a Gradle task that serves the performance reports over HTTP
omit 90f499f32f05 [improve][test] Fix the description of the renderHdrHistograms task
omit b918ec72a0a8 [improve][doc] Link the performance testing environment setup and the microbenchmarks
omit 55a89ef76dc0 [improve][build] Build the integration test image when it is out of date
omit ddcf77985845 [improve][test] Add a script that frees the Docker disk space that test runs use up
omit db11c9cb6498 [improve][test] Add a script that configures the host for performance testing and restores it afterwards
omit 342d6dd82f84 [improve][test] Set environment variables of the producer and consumer containers from the scenario
omit fa3e345949a5 [improve][test] Cut the cool-down before the measurement out of the throughput and backlog charts
omit b264d2bf7acb [improve][test] Signal the measurement start over HTTP instead of coordination files
omit f7ca35c4421a [improve][test] Report the host's CPU temperature and throttling, and cool down between runs
omit 5f3b2426aae6 [improve][test] Use AssertJ assertions in the performance launcher and report tool tests
omit b48aae1826fe [improve][test] Drop the delivery-after-publish estimate from the run report
omit 812a8e9d2f39 [improve][test] Name the end-to-end latency rows after their application
omit 605970fcbee2 [improve][test] Name each consumer application's output directory after its subscription
omit 2037fcc6230b [improve][test] Name each consumer application after its subscription in the run report and latency charts
omit f07021cbf66c [improve][test] Name the latency chart lines briefly
omit 5d13af282f58 [improve][test] Draw publish latency dotted, as the producers are in the throughput chart
omit 037b6288b3e3 [improve][test] Plot latency by percentile and over time per application with XChart, log latency every second, and write .hgrm percentile distributions
omit 5a84f1b09011 [improve][test] Leave the 0.0% label off the latency chart's baseline
omit 681ee9614044 [improve][test] Put the latency chart's axis labels right next to the axis
omit 8e91089c0d61 [improve][test] Put the latency chart's axis labels next to their panels, and the panels further apart
omit 7c6566dac4fc [improve][test] Move the performance reports and renderers to a report-tool module, and serve reports with their text files as text
omit 67754b945492 [improve][test] Count only Pulsar and BookKeeper frames as the application in profiles
omit 1ac898bf7984 [improve][test] Link a profile's files from tables with descriptive link texts, and its recordings from the run report
omit 7c611a3ca514 [improve][test] List a run's files in a collapsed section at the end of its report, and save container logs as text
omit a3a3c708804b [improve][test] Say which run each chart shows, link each run's files, and leave stacks without an application frame out of the app-root flame graphs
omit 262428e7c1e6 [improve][test] Write each launcher run to its own directory by day, branch and name, and record its provenance
omit b78c64566e97 [improve][test] Hide BookKeeper and Pulsar dispatch frames in the off-CPU digest and app-root flame graphs, abbreviate Java names on the digest page
omit 801a96643038 [improve][test] Follow the jonoffcpu 0.7.0 digest options and wording, end the backlog maximum at the producers' finish
omit 24d6b5c0b1cc [improve][test] Name only the rendered views, lead the run report with its profiles, bound the backlog maximum to the measurement, tidy the latency chart
omit 39c12a8cecd3 [improve][test] Give headings ids in the launcher's HTML report pages
omit 1ba8bb685066 [improve][test] Treat Jetty's reserved threads waiting for a job as idle in off-CPU profiles
omit 4cc75d416cec [improve][test] Write a run report with throughput, latency and sampled backlog, rendered to HTML
omit 17efe139ab6e [improve][test] Write a profile report, app-rooted off-CPU graphs and heatmaps for profiled runs
omit f00ba073d6f6 [improve][test] Upgrade jonoffcpu to 0.7.0
omit 0973a4917133 [improve][test] Use jonoffcpu 0.6.0 from its new home
omit 212ef4749bf1 [improve][doc] Document profiling performance scenarios with jonoffcpu
omit 28e14bb41b6c [improve][test] Render the configured flame graph views of profiled recordings
omit b408684cb729 [improve][test] Read the off-CPU idle-wait patterns from a resource file
omit a44b49f412c1 [improve][test] Render off-CPU flame graphs with and without idle waits
omit d8e65c6bc8ab [improve][test] Profile IoT performance scenarios with the jonoffcpu off-CPU agent
add b5a667602d1a [improve][broker] trace topic loading latency and timeout states (#26666)
add 986853d6e5e4 [fix][client] Don't close a shared DNS resolver group's address resolver when a client closes (#26723)
add d3978ac1f657 [feat][test] Add a Docker-based performance testing framework with profiling, metrics and AI agent support (#26714)
add 22a83855e7fc [improve][test] Add a launcher scenario for 500 producers and a 20-member Key_Shared subscription
add 6f7244642002 [improve][test] Name the Key_Shared 500x20 runs after their scenario files
add b9f9822cd9fd [improve][test] Move the Key_Shared 500x20 scenarios to the restructured scenario settings
add 1d34213ee7b5 [improve][test] Keep the Key_Shared 500x20 scenario saturating after the high-rate scenario got a rate limit
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (beb39f3df0d2)
\
N -- N -- N refs/heads/lh-key-shared-500x20-scenario (1d34213ee7b5)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../bookkeeper/mledger/impl/ManagedCursorImpl.java | 11 +-
.../mledger/impl/ManagedLedgerFactoryImpl.java | 5 +-
.../bookkeeper/mledger/impl/ManagedLedgerImpl.java | 23 +-
.../pulsar/broker/service/BrokerService.java | 216 +++++++---------
.../pulsar/broker/service/TopicLoadingContext.java | 141 ++++++----
.../broker/service/TopicLoadingTracePoints.java | 46 ++++
.../service/persistent/MessageDeduplication.java | 4 +-
.../broker/service/persistent/PersistentTopic.java | 6 +-
.../pulsar/broker/service/BrokerServiceTest.java | 144 +++++++++--
.../buffer/TopicTransactionBufferTest.java | 3 +-
.../pulsar/client/impl/PulsarClientImpl.java | 4 +-
.../pulsar/client/impl/PulsarClientImplTest.java | 36 +++
.../apache/pulsar/common/util/LatencyTracer.java | 288 ++++++++++++++++++---
.../pulsar/common/util/LatencyTracerTest.java | 182 ++++++++++---
14 files changed, 851 insertions(+), 258 deletions(-)
create mode 100644 pulsar-broker/src/main/java/org/apache/pulsar/broker/service/TopicLoadingTracePoints.java