You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "nzw921rx (via GitHub)" <gi...@apache.org> on 2026/07/30 08:53:18 UTC

[I] [Umbrella][Zeta] Establish a Long-Term Benchmark and Performance Regression Framework [seatunnel]

nzw921rx opened a new issue, #11598:
URL: https://github.com/apache/seatunnel/issues/11598

   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   
   
   ### Search before asking
   
   - [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
   
   
   ### Describe the proposal
   
   ## Background
   
   The [Apache SeaTunnel 3.0 Roadmap](https://github.com/apache/seatunnel/issues/11021) introduces a number of substantial and performance-sensitive changes to the Zeta engine.
   
   Representative changes affecting the core execution path include:
   
   - [#10800](https://github.com/apache/seatunnel/pull/10800): introduces timer scheduling, `FlushSignal` propagation, and signal handling across the Source, intermediate queue, Transform, and Sink execution path.
   - [#10572](https://github.com/apache/seatunnel/pull/10572): adds real-time task busyness, queue occupancy, backpressure, and processing-time observability across Source, Transform, Sink, and intermediate queues.
   - [#10491](https://github.com/apache/seatunnel/pull/10491): instruments the end-to-end record path with tracing stages from source emission to sink completion.
   - [#10306](https://github.com/apache/seatunnel/pull/10306): introduces row-level error handling and Dead Letter Queue support into the Transform and Sink execution paths.
   - [#10600](https://github.com/apache/seatunnel/pull/10600): adds table-level fault isolation involving multi-table writers, physical plans, and `JobMaster`.
   - [#10812](https://github.com/apache/seatunnel/pull/10812): introduces a StateStore abstraction across runtime state, checkpoint state, metrics snapshots, and engine services.
   - [#10860](https://github.com/apache/seatunnel/pull/10860): adds observability metrics for Zeta engine state stores.
   - [#10489](https://github.com/apache/seatunnel/pull/10489): adds lifecycle hooks and execution context support to the Zeta SQL UDF path.
   
   These changes are valuable for SeaTunnel 3.0, but they also make one existing gap increasingly visible: we currently have no standardized and reproducible way to evaluate their impact on Zeta engine performance.
   
   ## Problems
   
   ### 1. We cannot quantify the performance impact of major engine changes
   
   SeaTunnel 3.0 contains changes across task execution, intermediate queues, Transform and Sink lifecycle, tracing, state management, and multi-table execution.
   
   Correctness tests can verify whether these features work, but they cannot tell us:
   
   - whether throughput improved or regressed;
   - whether end-to-end latency changed;
   - whether CPU, memory, or GC costs increased;
   - whether a new control path adds overhead when enabled or disabled;
   - whether an optimization simply moves the bottleneck to another stage.
   
   Without a stable baseline, we cannot evaluate the cumulative performance impact of the 3.0 engine changes.
   
   ### 2. Zeta cannot provide evidence-based performance expectations
   
   Without a standard benchmark environment and workload definition, Zeta cannot provide credible and repeatable performance results across commits or releases.
   
   Individual test results produced with different hardware, JVM parameters, data volumes, payloads, parallelism, connectors, or measurement methods are not directly comparable.
   
   This makes it difficult for the project to clearly explain:
   
   - what level of throughput Zeta can provide under a defined workload;
   - how performance changes between releases;
   - which scenarios Zeta performs well in;
   - what performance cost is introduced by enabling a specific engine feature.
   
   ### 3. Code review alone cannot validate performance-sensitive optimizations
   
   Some engine changes involve lock scope, checkpoint locking, concurrent queues, Disruptor usage, scheduling, serialization, backpressure, or thread coordination.
   
   During review, we can reason about correctness and identify possible concurrency risks, but we usually cannot determine whether a change actually improves performance.
   
   For example, reducing one lock scope may improve concurrency, but it may also increase context switching, allocation, queue contention, or downstream pressure. Without before-and-after benchmark evidence, reviewers cannot reliably determine whether the change improves or reduces overall engine performance.
   
   ### 4. We cannot track whether Zeta is becoming faster or slower over time
   
   Zeta is continuously evolving, but the project currently has no long-term performance history.
   
   As a result, we cannot answer a basic question:
   
   > After months or years of engine development, is Zeta actually becoming faster, more resource-efficient, and more stable, or are performance regressions accumulating unnoticed?
   
   A single benchmark result is not enough. We need stable workloads, machine-readable results, and historical comparisons across commits and releases.
   
   ## Proposal
   
   Establish a long-term, standardized benchmark and performance regression framework for the Zeta engine.
   
   The framework should gradually provide:
   
   1. reproducible benchmark workloads for the core Zeta execution path;
   2. fixed and documented environment assumptions;
   3. comparable throughput, latency, CPU, memory, GC, queue, and backpressure metrics;
   4. machine-readable benchmark results;
   5. before-and-after comparison for engine pull requests;
   6. historical performance trends across commits and releases;
   7. specialized benchmarks for performance-sensitive engine components where necessary.
   
   The framework should be developed incrementally. The initial implementation should remain intentionally small and focus on establishing one reliable Zeta baseline before expanding into more scenarios.
   
   ## Phase 1
   
   The first phase should provide:
   
   - a real Zeta runtime rather than an embedded or mocked execution path;
   - deterministic in-memory benchmark Source and Sink implementations;
   - at least one core `Source -> Transform -> Sink` workload;
   - fixed data volume, payload size, parallelism, JVM parameters, and runtime configuration;
   - explicit warm-up and measurement stages;
   - repeated executions with a documented aggregation method;
   - machine-readable JSON results;
   - execution against the `dev` branch first;
   - no automatic PR blocking until variance and false-positive risks are understood.
   
   The implementation tracked by [#11580](https://github.com/apache/seatunnel/issues/11580) can serve as the first concrete delivery under this long-term initiative.
   
   ## Long-Term Goal
   
   The long-term goal is not only to run performance tests, but to establish a trusted evaluation framework for Zeta engine development.
   
   For every performance-sensitive engine change, contributors and reviewers should eventually be able to answer:
   
   - What scenario is affected?
   - What was the baseline?
   - What changed after the patch?
   - Is the result outside the normal benchmark variance?
   - Did throughput, latency, or resource usage improve?
   - Did the optimization introduce a regression in another part of the execution path?
   
   Once the framework has accumulated enough stable data, it may gradually evolve into a reliable performance regression signal for pull requests and releases.
   
   ### Task list
   
   ### Phase 1: Initial benchmark framework and baseline
   
   - [ ] #11580
     - [ ] Define the initial architecture and repository boundaries for the Zeta benchmark framework
     - [ ] Bootstrap the independent `seatunnel-benchmarks` project
     - [ ] Support selecting the tested SeaTunnel version, distribution, image, and commit
     - [ ] Implement a deterministic `BenchmarkSource` based only on the public SeaTunnel API
     - [ ] Implement a lightweight `PassThroughTransform`
     - [ ] Implement a `BlackholeSink` that does not introduce external I/O overhead
     - [ ] Implement the initial Zeta benchmark runner using a real Zeta runtime
     - [ ] Add the first `BenchmarkSource -> PassThroughTransform -> BlackholeSink` baseline workload
     - [ ] Define the benchmark scenario configuration model
     - [ ] Define warm-up, measurement, repetition, timeout, and result aggregation rules
     - [ ] Define a machine-readable JSON result format
     - [ ] Record the tested commit, environment, JVM parameters, workload parameters, and runtime configuration in every result
     - [ ] Document how to build and run the first benchmark locally
     - [ ] Run the initial benchmark against the `dev` branch without using it as a PR gate
   
   ### Phase 2: Core Zeta workload matrix
   
   - [ ] Add core record-processing throughput benchmarks
     - [ ] `Source -> Sink`
     - [ ] `Source -> PassThroughTransform -> Sink`
     - [ ] Single Transform and chained Transform workloads
     - [ ] Narrow-row, wide-row, string-heavy, and binary-heavy workloads
   
   - [ ] Add intermediate queue and backpressure benchmarks
     - [ ] Fast Source with a controllable slow Sink
     - [ ] Queue occupancy and backpressure propagation
     - [ ] Blocking queue and Disruptor execution paths
     - [ ] Different parallelism and queue-capacity configurations
   
   - [ ] Add serialization and data-transfer benchmarks
     - [ ] `SeaTunnelRow` serialization and deserialization
     - [ ] Cross-task record transfer
     - [ ] Different payload sizes and field counts
   
   - [ ] Add job lifecycle benchmarks
     - [ ] Job parsing and DAG generation
     - [ ] Physical plan generation
     - [ ] Task deployment
     - [ ] Job startup and shutdown
   
   - [ ] Add state and checkpoint benchmarks
     - [ ] Checkpoint duration and alignment cost
     - [ ] StateStore read and write performance
     - [ ] Checkpoint restore performance
     - [ ] State-store growth and cleanup behavior
   
   ### Phase 3: Engine-internal microbenchmarks
   
   - [ ] Define the Zeta internal microbenchmark development model
     - [ ] Keep benchmark implementations that depend on internal classes in the corresponding `seatunnel-engine-*` module under `src/test`
     - [ ] Add ordinary unit tests to ensure each internal benchmark remains runnable
     - [ ] Keep only thin JMH executors in the `seatunnel-benchmarks` project
     - [ ] Avoid exposing internal Engine classes as public APIs only for benchmarking
   
   - [ ] Add initial Engine microbenchmarks
     - [ ] Lock and synchronization paths
     - [ ] Intermediate queue operations
     - [ ] Disruptor event publication and consumption
     - [ ] Record serialization
     - [ ] DAG and physical-plan generation
     - [ ] Task scheduling and deployment
     - [ ] StateStore operations
     - [ ] Checkpoint coordination
   
   - [ ] Produce an executable benchmark JAR
     - [ ] Support selecting individual benchmarks
     - [ ] Support benchmark parameters
     - [ ] Support JMH profilers and flame-graph generation
     - [ ] Support JSON and CSV output
   
   ### Phase 4: Automation and historical performance tracking
   
   - [ ] Add a dedicated benchmark execution environment
     - [ ] Use fixed hardware, JDK, JVM parameters, container configuration, and operating-system settings
     - [ ] Prevent unrelated workloads from running on the benchmark machine
     - [ ] Separate build time, startup time, warm-up time, and steady-state measurement time
   
   - [ ] Add scheduled benchmark execution for the `dev` branch
     - [ ] Build the tested SeaTunnel commit
     - [ ] Run the standard benchmark suite
     - [ ] Upload raw results and environment metadata
     - [ ] Retain logs and profiling artifacts
   
   - [ ] Add historical benchmark result storage and visualization
     - [ ] Track results by commit and release
     - [ ] Display throughput, latency, CPU, memory, GC, and backpressure trends
     - [ ] Mark environment or workload-definition changes
     - [ ] Prevent incomparable results from being placed in the same trend line
   
   ### Phase 5: Pull request comparison and regression detection
   
   - [ ] Add base-versus-head benchmark comparison
     - [ ] Run the same workloads against the base and head commits
     - [ ] Repeat measurements and report median and variance
     - [ ] Report throughput, latency, and resource-usage differences
     - [ ] Produce a machine-readable and human-readable comparison report
   
   - [ ] Add non-blocking benchmark reports for selected Engine pull requests
     - [ ] Trigger only for performance-sensitive Zeta changes
     - [ ] Publish results without blocking merges during the initial observation period
     - [ ] Mark unstable results as `INCONCLUSIVE`
     - [ ] Collect enough historical data to understand normal variance
   
   - [ ] Define the performance regression policy
     - [ ] Define benchmark stability requirements
     - [ ] Define acceptable variance and regression thresholds
     - [ ] Define rerun and investigation rules
     - [ ] Define benchmark ownership and failure-triage responsibilities
   
   - [ ] Evaluate introducing a Zeta performance regression gate
     - [ ] Enable it only for stable workloads
     - [ ] Start with a small set of trusted Engine benchmarks
     - [ ] Require evidence and manual review before treating a result as a regression
   
   ### Are you willing to submit PR?
   
   - [x] Yes I am willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [STIP-32][Umbrella][Zeta] Establish a Long-Term Benchmark and Performance Regression Framework [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on issue #11598:
URL: https://github.com/apache/seatunnel/issues/11598#issuecomment-5203022761

   Thanks for the well-motivated proposal. Since this is a STIP, please ensure the design is also shared on the dev@ mailing list for broader visibility and consensus before implementation begins.
   
   Could you clarify the intended execution environment for benchmarks (dedicated hardware vs. GitHub Actions runners) and how run-to-run variance will be controlled? This strongly affects whether regression detection thresholds are reliable.
   
   Suggest defining the initial baseline (e.g., current 2.x Zeta release) and a small set of representative workloads (single-table, multi-table, checkpoint-heavy) as the first subtasks, so each 3.0 PR listed here can be measured against a stable reference.
   
   <!-- streview-comment:57 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [STIP-32][Umbrella][Zeta] Establish a Long-Term Benchmark and Performance Regression Framework [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #11598:
URL: https://github.com/apache/seatunnel/issues/11598#issuecomment-5137560662

   Thanks for writing this as a long-term umbrella/STIP proposal.
   
   I think this belongs on the design/STIP track rather than being treated as only a small feature request. The important part is that the proposal separates the first runnable baseline from the later regression-gate ambition. That boundary should stay firm: #11580 can be the first concrete implementation slice, but the project should not turn benchmark results into a PR-blocking signal until the workload variance, hardware isolation, rerun policy, and false-positive risk are understood from real data.
   
   For Phase 1, I would keep the acceptance criteria very explicit:
   
   1. run against a real Zeta runtime, not only an embedded/mock path;
   2. use deterministic benchmark Source/Sink implementations so external I/O does not dominate the first result;
   3. record the tested SeaTunnel commit, JVM/runtime parameters, workload size, parallelism, warm-up/measurement rules, and environment metadata in every JSON result;
   4. keep the initial workload small enough that reviewers can understand and reproduce it;
   5. document that Phase 1 is observational only and not a merge gate.
   
   For the later phases, one design point is especially important: benchmarks that need engine-internal classes should not force those internals into public APIs just for measurement. Keeping internal microbenchmarks close to their owning engine modules, with only thin runnable wrappers where needed, will make the framework easier to maintain.
   
   So my recommendation is to keep this issue open as the long-term benchmark framework tracker, link #11580 as the first delivery slice, and require each benchmark PR to state which phase and which workload boundary it implements.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [STIP-32][Umbrella][Zeta] Establish a Long-Term Benchmark and Performance Regression Framework [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #11598:
URL: https://github.com/apache/seatunnel/issues/11598#issuecomment-5509300689

   Thanks for the architecture summary. It is a useful Phase 1 boundary: benchmark infrastructure remains opt-in, measures production paths, records normalized artifacts, and is not a blocking regression gate. The linked delivery history supports that reading: #11671, #11986, #12021, and #12026 are merged, while the state storage and recovery slice in #12044 is still open. Please keep this STIP open as the design and delivery tracker until that remaining slice has completed review and merged; any future gate proposal still needs a controlled runner, variance data, rerun policy, and ownership before it can become enforcement.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [STIP-32][Umbrella][Zeta] Establish a Long-Term Benchmark and Performance Regression Framework [seatunnel]

Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx commented on issue #11598:
URL: https://github.com/apache/seatunnel/issues/11598#issuecomment-5507879596

   ## Architecture Overview
   
   This section summarizes the final architecture of the Zeta benchmark framework after #11671, #11986, #12021, #12026, and #12044.
   
   The goal is not to benchmark every Engine class.
   
   The framework provides a reusable way to:
   
   > measure a real performance path → compare revisions → diagnose a change → verify an optimization.
   
   ---
   
   ### 1. Overall architecture
   
   The benchmark framework lives in the main SeaTunnel repository as an opt-in `seatunnel-benchmarks` module.
   
   It directly exercises production Zeta implementations while remaining outside the normal build and runtime classpath.
   
   ```mermaid
   flowchart TB
       CI["Execution<br/>GitHub Actions / Local"]
       JMH["JMH benchmark layer"]
       Env["Benchmark environments<br/>& deterministic fixtures"]
       Zeta["Production Zeta code"]
       Result["Results & comparison"]
       Diag["Diagnostics"]
   
       CI --> JMH
       JMH --> Env
       Env --> Zeta
   
       JMH --> Result
       Zeta --> Result
   
       Result --> Diag
   ```
   
   The main boundary is:
   
   ```text
   benchmark-only infrastructure
           ↓
   production implementation
   ```
   
   Benchmark convenience should not leak into production APIs.
   
   When internal access is required, prefer a narrowly scoped benchmark-side bridge rather than adding an Engine API only for testing.
   
   ---
   
   ### 2. Benchmark coverage
   
   The current suite covers the major performance-sensitive Zeta paths.
   
   ```mermaid
   flowchart LR
       Record["Record processing"]
   
       Row["SeaTunnelRow"]
       Queue["Intermediate Queue"]
       Pipeline["Complete Pipeline"]
   
       Checkpoint["Checkpoint"]
   
       Coord["Coordination"]
       CPStore["Persistence"]
   
       State["Engine State"]
   
       IMap["IMap / StateStore"]
       DAG["Job DAG"]
       WAL["WAL / Recovery"]
   
       Record --> Row
       Record --> Queue
       Record --> Pipeline
   
       Checkpoint --> Coord
       Checkpoint --> CPStore
   
       State --> IMap
       State --> DAG
       State --> WAL
   ```
   
   Current implementations:
   
   | Area | Benchmark |
   | --- | --- |
   | Row operations | `SeaTunnelRowBenchmark` |
   | Record handoff | `IntermediateQueueBenchmark` |
   | End-to-end data path | `SeaTunnelPipelineBenchmark` |
   | Checkpoint coordination | `CheckpointingTimeBenchmark` |
   | Checkpoint persistence | `CheckpointStorageBenchmark` |
   | Job/metrics state | `IMapJobStorageBenchmark` |
   | DAG persistence | `IMapDagStorageBenchmark` |
   | WAL and recovery | `IMapWalStorageBenchmark` |
   
   New benchmark areas should be added only when an actual performance question requires them.
   
   ---
   
   ### 3. Measurement model
   
   A common rule across the framework is:
   
   > Setup and validation should not accidentally become part of the measured operation.
   
   ```mermaid
   flowchart LR
       Setup["Prepare environment<br/>and fixtures"]
       Measure["Measured production path"]
       Verify["Validate correctness<br/>and persistence"]
       Cleanup["Cleanup"]
   
       Setup --> Measure --> Verify --> Cleanup
   
       style Measure stroke-width:4px
   ```
   
   The exact measurement boundary depends on the question.
   
   For example:
   
   ```mermaid
   flowchart TB
       Pipeline["Pipeline benchmark<br/>Job submission → complete data path → job completion"]
   
       CP["Checkpoint benchmark<br/>Trigger → barrier/snapshot/ACK → persistence → completion"]
   
       Storage["Storage benchmark<br/>Prepared production fixture → target persistence operation"]
   
       Queue["Queue benchmark<br/>One production record handoff"]
   
   ```
   
   This prevents unrelated startup or fixture cost from hiding the path being investigated.
   
   ---
   
   ### 4. Pipeline workload model
   
   The full Pipeline benchmark uses deterministic in-memory Source, Transform, and Sink implementations over a real embedded Zeta runtime.
   
   ```mermaid
   flowchart LR
       Source["BenchmarkSource<br/>open-loop schedule"]
       Queue1["Zeta runtime"]
       Transform["Transform<br/>optional"]
       Queue2["Zeta runtime"]
       Sink["BenchmarkSink"]
       Metrics["Throughput<br/>P50/P95/P99<br/>latency growth"]
   
       Source --> Queue1 --> Transform --> Queue2 --> Sink
       Sink --> Metrics
   ```
   
   The Source follows an absolute open-loop schedule.
   
   When Zeta cannot keep up, the Source schedule continues advancing, so backlog appears as increasing end-to-end latency instead of being hidden by a slower producer.
   
   Correctness is checked before interpreting performance:
   
   ```text
   expected rows == processed rows
   Transform checksum is valid
   ```
   
   ---
   
   ### 5. Persistent-state benchmark model
   
   Storage benchmarks use real Zeta StateStore, Hazelcast IMap, `FileMapStore`, WAL, and checkpoint storage with local files.
   
   Persistent mutation benchmarks use **fixed logical work**, rather than “as many operations as possible”.
   
   ```mermaid
   flowchart LR
       Fixture["Prepare identical state"]
       A["Revision A<br/>100 operations"]
       B["Revision B<br/>100 operations"]
       Compare["Compare time/op"]
   
       Fixture --> A --> Compare
       Fixture --> B --> Compare
   ```
   
   This is important because a throughput-style storage benchmark could otherwise make a faster revision create more:
   
   ```text
   keys
   WAL records
   files
   mutation history
   ```
   
   and therefore change its own workload.
   
   Fixed batches plus `OperationsPerInvocation` keep storage pressure comparable.
   
   ---
   
   ### 6. Stable core vs investigation benchmarks
   
   Not every benchmark belongs in scheduled execution.
   
   ```mermaid
   flowchart TB
       All["All benchmark scenarios"]
   
       Core["Stable core suite<br/>scheduled"]
       Manual["Focused benchmarks<br/>manual"]
       Expensive["Expensive diagnostics<br/>manual only"]
   
       All --> Core
       All --> Manual
       All --> Expensive
   ```
   
   `benchmarks_core` contains representative, stable, bounded workloads.
   
   Large parameter matrices such as deep WAL recovery remain on-demand.
   
   The core suite should stay small enough that scheduled runs remain practical.
   
   ---
   
   ### 7. Execution and comparison
   
   Normal execution produces raw and normalized artifacts rather than only console output.
   
   ```mermaid
   flowchart LR
       Run["JMH run"]
       Raw["Raw JMH JSON"]
       Pipe["Pipeline JSON"]
       Normalize["Normalized report"]
       Summary["Markdown summary"]
   
       Run --> Raw
       Run --> Pipe
   
       Raw --> Normalize
       Pipe --> Normalize
   
       Normalize --> Summary
   ```
   
   The normalized format records:
   
   ```text
   benchmark
   parameters
   score / direction
   raw samples
   commit
   JDK / JVM
   runner
   CPU / memory
   workload identity
   correctness
   ```
   
   This keeps the current framework usable even if historical storage or visualization is added later.
   
   ---
   
   ### 8. PR comparison
   
   PR comparison uses the same worker and alternates revisions:
   
   ```mermaid
   flowchart LR
       B1["Baseline"]
       C1["Candidate"]
       C2["Candidate"]
       B2["Baseline"]
   
       B1 --> C1 --> C2 --> B2
   ```
   
   The report compares the median of the two baseline runs with the median of the two candidate runs.
   
   Metric direction is normalized so:
   
   > positive change means improvement
   
   for both higher-is-better throughput and lower-is-better latency/time metrics.
   
   The result remains observational because GitHub-hosted runner performance is not sufficiently controlled for a hard regression gate.
   
   ---
   
   ### 9. Measurement and diagnostics are separate
   
   Profiling is used to explain a benchmark result, not to produce the benchmark baseline itself.
   
   ```mermaid
   flowchart LR
       Benchmark["Normal benchmark"]
       Change["Unexpected difference"]
   
       CPU["CPU"]
       Wall["Wall"]
       Lock["Lock"]
       GC["GC / Allocation"]
       JFR["JFR"]
   
       Root["Root cause"]
       Fix["Production optimization"]
       Verify["Benchmark again"]
   
       Benchmark --> Change
   
       Change --> CPU
       Change --> Wall
       Change --> Lock
       Change --> GC
       Change --> JFR
   
       CPU --> Root
       Wall --> Root
       Lock --> Root
       GC --> Root
       JFR --> Root
   
       Root --> Fix --> Verify
   ```
   
   Diagnostics are intentionally kept in a separate workflow because profiler overhead changes benchmark behavior.
   
   Supported diagnostics currently include:
   
   ```text
   CPU
   Wall clock
   Lock contention
   GC / allocation
   JFR
   ```
   
   ---
   
   ### 10. How the framework should evolve
   
   The framework is now intended to grow **from real performance questions**, rather than from a predefined list of classes.
   
   ```mermaid
   flowchart LR
       Change["Engine change<br/>or suspicious result"]
       Existing{"Existing benchmark<br/>can measure it?"}
       Use["Use existing benchmark"]
       Add["Add focused benchmark"]
       Diagnose["Compare / diagnose"]
       Optimize["Optimize Zeta"]
   
       Change --> Existing
   
       Existing -->|Yes| Use
       Existing -->|No| Add
   
       Use --> Diagnose
       Add --> Diagnose
   
       Diagnose --> Optimize
   ```
   
   Examples of future demand-driven areas may include:
   
   - checkpoint coordinator fan-in;
   - serialization or cross-task transfer;
   - additional state/recovery paths;
   - scheduler hot paths;
   - long-running stability workloads.
   
   They are not mandatory unfinished parts of STIP-32.
   
   ---
   
   ### 11. Future history and regression gate
   
   Long-term result storage and a blocking regression gate remain optional future evolution.
   
   ```mermaid
   flowchart LR
       Today["Current framework<br/>versioned result artifacts"]
       History["Long-term storage"]
       Trend["Performance history"]
       Signal["Regression signal"]
       Gate["Optional blocking gate"]
   
       Today -.-> History
       History --> Trend
       Trend --> Signal
       Signal -.-> Gate
   ```
   
   The current result format already contains the data required to build this later.
   
   The missing requirement is mainly operational:
   
   ```text
   controlled runner
   stable environment
   historical variance
   rerun policy
   ownership
   false-positive control
   ```
   
   Until those requirements exist, the benchmark framework should remain an evidence and diagnosis tool rather than a blocking CI gate.
   
   ---
   
   ## Design summary
   
   The final architecture follows a few simple rules:
   
   1. **Measure production paths, not mocks.**
   2. **Keep setup and validation outside the measurement unless intentionally measured.**
   3. **Validate correctness before trusting performance.**
   4. **Use fixed logical work for persistent mutation benchmarks.**
   5. **Keep the scheduled core suite small and stable.**
   6. **Separate benchmark measurement from profiling diagnostics.**
   7. **Do not change production APIs only for benchmark convenience.**
   8. **Add new benchmarks when a real performance question requires them.**
   
   The framework should now primarily be used to improve Zeta rather than continuously expand benchmark coverage.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Umbrella][Zeta] Establish a Long-Term Benchmark and Performance Regression Framework [seatunnel]

Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx commented on issue #11598:
URL: https://github.com/apache/seatunnel/issues/11598#issuecomment-5128709509

   Firstly, I believe this is a long-term STIP plan, but its importance lies in outputting a reliable performance evaluation standard that can be recognized by all members of the community team, so that the engine can continue to steadily move in the right direction.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [STIP-32][Umbrella][Zeta] Establish a Long-Term Benchmark and Performance Regression Framework [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo closed issue #11598: [STIP-32][Umbrella][Zeta] Establish a Long-Term Benchmark and Performance Regression Framework
URL: https://github.com/apache/seatunnel/issues/11598


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org