You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "zhangshenghang (via GitHub)" <gi...@apache.org> on 2026/07/19 14:18:10 UTC
[PR] [Fix][Zeta] Reset metrics after pipeline recovery [seatunnel]
zhangshenghang opened a new pull request, #11495:
URL: https://github.com/apache/seatunnel/pull/11495
## What does this PR do?
Resets runtime metric snapshots before a failed Zeta pipeline is restored.
## Why is this change needed?
Pipeline retries reuse stable task locations. Retaining snapshots from the failed attempt caused `SourceReceivedCount`, `SinkWriteCount`, and queue metrics to continue from stale values, so finished-job metrics could include work from canceled attempts after worker failover.
## How was it tested?
- Added `StateTransitionCleanupTest#testPipelineRestoreClearsMetricsFromTheFailedAttempt`.
- `./mvnw -pl seatunnel-engine/seatunnel-engine-server -DskipITs -Dskip.spotless=true -Dtest=StateTransitionCleanupTest test`
- `./mvnw -q -pl seatunnel-engine/seatunnel-engine-server -DskipTests verify`
- `./mvnw -q -DskipTests verify`
Closes #11472.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Reset metrics after pipeline recovery [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11495:
URL: https://github.com/apache/seatunnel/pull/11495#issuecomment-6359789807
Thanks for the dev sync and for rerunning the failed jobs, @zhangshenghang, and thanks @SEZ9 for the per-finding recap. I re-checked live: the head is still `66f46aa371a7` (the one I last reviewed; the newest commit is a pure `dev` merge), so my source conclusion is unchanged and there is no new source-side finding from me.
On the CI facts: `Build` is still FAILURE on this head, and I agree the failures do not come from this diff, which only touches the pipeline-restore metrics cleanup. One small correction to the diagnosis so the next step is the right one: the identical failure set on both attempts does not point to a random runner problem. The `PreconditionViolationException` / empty Testcontainers invocation context provider signature is the known E2E infrastructure bug tracked in #12889: with a Zeta-only engine scope, suites annotated with `@DisabledOnContainer` get zero containers and JUnit fails every test in the class. Its fix, #12895 (`[Fix][E2E] Skip suites when all selected containers are disabled`), is still open, which is why a rerun reproduces it deterministically. I could only confirm that signature from the logs I had access to; I could not re-read the HBase and MySQL-CDC job logs for this head (the log endpoint returned errors), so I am not claiming they share the same cause.
Minimal next step: please do not add unrelated patches here and no further blind reruns are needed. Once #12895 lands in `dev`, sync `dev` once and rerun only the failed jobs (`gh run rerun <run-id> --failed`), then drop a note here. If any job still fails after that with a different error, post its first error and I will check it against the diff.
On SEZ9's F4-F7 recap: they stay non-blocking from my side too; a short note in the PR that they are deferred (or a quick pickup if convenient) is enough. The remaining gates are a green `Build` on the final head and the final maintainer approval. I will recheck as soon as a new commit or a green run is posted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Reset metrics after pipeline recovery [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11495:
URL: https://github.com/apache/seatunnel/pull/11495#issuecomment-5129169951
Thanks for the update. I rechecked the current head `4729fe2ecf3d` against the latest `dev`. This branch is currently `diverged` (`ahead_by=1`, `behind_by=62`), and the current `Build` check is `FAILURE`: https://github.com/apache/seatunnel/runs/88202413330
The latest `dev` branch has a newer green `On pull request update` run (https://github.com/apache/seatunnel/actions/runs/30529407120, head `40700ad6241a`), so this CI signal is not very actionable on the stale PR head yet. The lowest-cost next step is to sync with the latest `dev` and rerun the failing check first. If it still fails after the sync, please share the new failing check link and I can help narrow it down against the updated head.
This follow-up records the current CI/upstream-sync gate only; it should not be read as blaming the PR diff for the stale-head Build failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Reset metrics after pipeline recovery [seatunnel]
Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang commented on PR #11495:
URL: https://github.com/apache/seatunnel/pull/11495#issuecomment-6136232020
Thanks @DanielLeens — Issue 1 is addressed on head `f45b6de6`: `clearPipelineMetricsForRestore` now calls `cleanTaskGroupContext(pipelineLocation)` **before** `seaTunnelServer.removeMetrics(...)`, so the failed attempt's worker-side task-group contexts are wiped first and can no longer re-populate the metrics store after the reset. Both steps keep the existing best-effort semantics (a cleanup failure logs a warning and never blocks the restore).
Also noted your pointer on the ES IT — `#12711` stabilising index visibility polling on `dev` explains the `PIT/Scroll-slice` failures we parked; once that lands in our sync base those lanes should go green on their own.
`StateTransitionCleanupTest` is 11/11 green locally with the change. New head is synced with current `dev`; ready for re-review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Reset metrics after pipeline recovery [seatunnel]
Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang commented on PR #11495:
URL: https://github.com/apache/seatunnel/pull/11495#issuecomment-6321843600
Heads-up on the failing `Build` on this head: it is inherited from `dev`, not caused by this PR.
- **unit-test (all 4 matrix jobs)**: `LLMBooleanOutputTest.readsLegacyFactoryWrapperWithValidationDisabled` fails with `InvalidClassException: AbstractMultiCatalogTransform ... serialVersionUID = 7090258061839605137 vs -3241928148724104003`. This PR touches nothing in `seatunnel-transforms-v2` (`AbstractMultiCatalogTransform` is byte-identical to `dev`), and dev's own push Build at `bab2d089c` fails with the exact same exception ([run 39536327852](https://github.com/apache/seatunnel/actions/runs/39536327852)). The upstream fix appears to be in flight at #12938.
- **engine-v2-it**: 207/208 pass; the single error is `BackpressureSlowSinkIT`, a known environment-flaky test unrelated to this change.
Once #12938 lands in `dev`, we will sync `dev` into this branch so the Build reflects it. No rerun was attempted since both failures are deterministic/inherited.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Reset metrics after pipeline recovery [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on code in PR #11495:
URL: https://github.com/apache/seatunnel/pull/11495#discussion_r3653052324
##########
seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/dag/physical/StateTransitionCleanupTest.java:
##########
@@ -97,6 +105,57 @@ void testPhysicalPlanIgnoresLateTransitionWhenJobStateAlreadyTerminal() throws E
Assertions.assertEquals(JobStatus.FAILED, planWithStateMaps.runningJobState.get(jobId));
}
+ @Test
+ void testPipelineRestoreClearsMetricsFromTheFailedAttempt() throws Exception {
+ long jobId = instance.getFlakeIdGenerator(Constant.SEATUNNEL_ID_GENERATOR_NAME).newId();
+ JobImmutableInformation jobImmutableInformation = mock(JobImmutableInformation.class);
+ JobConfig jobConfig = mock(JobConfig.class);
+ Map<String, Object> envOptions = new HashMap<>();
+ envOptions.put("job.retry.times", "1");
+ envOptions.put("job.retry.interval.seconds", "0");
+ when(jobImmutableInformation.getJobId()).thenReturn(jobId);
+ when(jobImmutableInformation.getJobConfig()).thenReturn(jobConfig);
+ when(jobConfig.getEnvOptions()).thenReturn(envOptions);
+
+ IMap<Object, Object> runningJobState =
+ nodeEngine.getHazelcastInstance().getMap("restoreMetricsState-" + jobId);
+ IMap<Object, Long[]> runningJobStateTimestamps =
+ nodeEngine.getHazelcastInstance().getMap("restoreMetricsStateTimestamps-" + jobId);
+ PipelineLocation pipelineLocation = new PipelineLocation(jobId, 1);
+ runningJobState.put(pipelineLocation, PipelineStatus.FAILED);
+
+ ExecutorService executorService = Executors.newSingleThreadExecutor();
+ try {
+ SubPlan subPlan =
+ new SubPlan(
+ 1,
+ 1,
+ System.currentTimeMillis(),
+ Collections.emptyList(),
+ Collections.emptyList(),
+ jobImmutableInformation,
+ executorService,
+ runningJobState,
+ runningJobStateTimestamps,
+ Collections.emptyMap());
+ JobMaster jobMaster = mock(JobMaster.class);
+ CheckpointManager checkpointManager = mock(CheckpointManager.class);
+ PhysicalPlan physicalPlan = mock(PhysicalPlan.class);
+ when(jobMaster.getCheckpointManager()).thenReturn(checkpointManager);
+ when(jobMaster.getPhysicalPlan()).thenReturn(physicalPlan);
+ subPlan.setJobMaster(jobMaster);
+
+ java.lang.reflect.Method restore =
Review Comment:
+1. The current head still uses the fully qualified java.lang.reflect.Method at line 148; a normal import is clearer and keeps the test consistent with the project's import style. This is non-blocking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Reset metrics after pipeline recovery [seatunnel]
Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang commented on PR #11495:
URL: https://github.com/apache/seatunnel/pull/11495#issuecomment-6135718219
Thanks for following up — quick correction so we're looking at the same state: the head you referenced (`4729fe2ecf3d`, July) has been superseded. The branch was synced with the latest `dev` on 2026-10-11 and the current head is `0ac0f2e5a0` (merge commit resolving the metrics/docs conflicts).
On that head the fork `Build` run (38112486866) completed with **76/81 jobs green**, including all unit tests and the engine lanes. The only failures were `elasticsearch-connector-it (11)`/`(17)` — the same 4 PIT/Scroll-slice cases failing on two consecutive attempts — which are environment-dependent and unrelated to this PR's Zeta metrics change (no ES code touched). We've parked them per the flake policy rather than burning reruns; happy to share logs if you want to help narrow them down against upstream.
So: upstream sync is already done, the compile/unit signal on the current head is green, and the remaining red is the ES IT environment family. Ready for re-review on `0ac0f2e5a0` whenever you have a moment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Reset metrics after pipeline recovery [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11495:
URL: https://github.com/apache/seatunnel/pull/11495#issuecomment-6210064113
Thanks @SEZ9 for the follow-up. To save a round trip, here is where each of your three points stands on `9c047310570519504bd5d37be03421b46b377dba`, as I read the code (I have not run a failover reproduction):
1. Worker cleanup and the re-report race. `SubPlan.prepareRestorePipeline` (`SubPlan.java:497`) now calls `JobMaster.clearPipelineMetricsForRestore`, which first sends `CleanTaskGroupContextOperation` to every owned slot whose worker is still a cluster member (per task group, failures logged and skipped), and only then removes the pipeline snapshots from the master store. If the clean fails on a worker that is still alive, that context is not removed, so it can still re-publish the old attempt's counters after the wipe. Both the Javadoc and the PR notes describe the race as a millisecond-scale window for an in-flight report, but neither states this failed-clean-on-a-live-worker case explicitly. I agree with you that it should be stated plainly so the PR does not over-claim a complete fix for #11472. @zhangshenghang, could you add one sentence for it to the PR description?
2. Counter semantics. The behavior change is listed in `docs/en/introduction/concepts/incompatible-changes.md` (and the `zh` twin) under "Zeta: runtime metric counters restart from zero after a pipeline failover restore", with a shorter note in `docs/{en,zh}/engines/zeta/rest-api-job-lifecycle.md`. It covers streaming jobs resumed after a failover. The PR description's Notes section also points to both places.
3. Tests. `testRestoreClearRunsTaskGroupCleanupBeforeMetricsRemoval` uses `InOrder` to prove clean-before-remove. `testRestoreClearRemovesMetricsFromTheStateStore` seeds the real `MetricsSnapshotStateStore` and asserts the restored pipeline's snapshots are gone while other pipelines are untouched. None of the tests simulates a late worker report from the old attempt, so the "late report does not reappear" assertion you describe is not covered. Given the best-effort design, I would treat that as a nice-to-have rather than a blocker, but I defer to you on whether you want it before approving.
CI: the `Build` check on this head is green now. I'll re-check as soon as the description update lands.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Reset metrics after pipeline recovery [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11495:
URL: https://github.com/apache/seatunnel/pull/11495#issuecomment-6228878317
Thanks for the detailed walkthrough of `9c047310570519504bd5d37be03421b46b377dba`. I have not gone through the diff myself yet, so I am treating the points below as to-be-confirmed rather than verified:
**F1 / F2** – If the restore path now sends `CleanTaskGroupContextOperation` to the owned slots on live members before removing the pipeline snapshots from the master store, that ordering addresses the core of both findings. As noted above, a clean that fails on a worker that is still alive leaves that `TaskGroupContext` in place, so it can re-publish the previous attempt's counters after the wipe. Please add one sentence stating that case explicitly in the PR description (and ideally the method Javadoc), so the change is presented as a best-effort fix rather than a complete one.
**F3** – If the behavior change is documented in the incompatible-changes page (en and zh) and the Zeta REST API job-lifecycle page as described, that covers my concern. I will confirm against the changed files.
**F4** – An `InOrder` test for clean-before-remove and a test seeding the real `MetricsSnapshotStateStore` are what I was asking for. The "late report from the old attempt does not reappear" scenario would still be uncovered; given the best-effort design I will not block on it, but if it is cheap to add I would welcome it here, otherwise a short follow-up note is fine.
**F5 / F6 / F7** – I do not see these covered in the thread. Could you confirm whether the `CALLS_REAL_METHODS` plus reflective field-injection scaffolding, the fully-qualified `java.lang.reflect.*` / `org.mockito.Mockito.*` references, and the duplication between `clearPipelineMetricsForRestore` and `removeMetricsContext` were touched in this head? Consolidating the two helpers and switching to imports should be quick; the test scaffolding can be a follow-up.
On where the clear is called in `prepareRestorePipeline`: if `reset()` has no other callers, the two placements behave the same today, and I am fine keeping the call alongside the other restore-path steps. That is a style choice and yours to make.
To summarize, before I approve: (1) the one-sentence failed-clean-on-live-worker note in the description/Javadoc, and (2) a quick status on F5–F7.
<!-- streview-comment:1817 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Reset metrics after pipeline recovery [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11495:
URL: https://github.com/apache/seatunnel/pull/11495#issuecomment-5167177604
Rechecked the latest head on August 3, 2026: the earlier failing CI signal I referenced is no longer the current fact. The latest head is now pending in CI again, so I do not want to leave the old failure guidance in place.
From Daniel's side I do not see a new code-side blocker on this head at the moment; the remaining gate for me is the current CI / merge state. I only have read-only review controls in this repository, so a maintainer with write access may still need to handle the final approval or dismissal step once the checks finish.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Reset metrics after pipeline recovery [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11495:
URL: https://github.com/apache/seatunnel/pull/11495#issuecomment-6207573843
Thanks for the update on `9c047310570519504bd5d37be03421b46b377dba`. CI is no longer a concern from my side; what remains before I can approve is closing out the earlier review points, which I have not yet re-verified against this head.
Could you confirm the current state of each of these, either in the PR description or in a reply here?
1. **Worker-side cleanup and the re-report race (F1, F2).** Does the restore path now send `CleanTaskGroupContextOperation` to surviving workers? If so, what happens when that operation fails on a still-alive worker, and what prevents a metrics report already in flight from re-populating the map before the new attempt deploys? If this case is not fully covered, please state the limitation plainly in the PR description so the change does not over-claim a complete fix.
2. **Counter semantics for streaming jobs (F3).** If counters now reset to zero on every pipeline retry by design, please add a short note in the docs / incompatible-changes section and reference it from the PR description. If that is not intended, the reset should be scoped.
3. **Test coverage (F4).** Do any of the new tests go beyond verifying a mock invocation? Ideally at least one test would use the real metrics store, exercise the worker re-report path, and assert that the restored attempt starts from zero and that a late report from the old attempt does not reappear.
Non-blocking, but worth addressing in the same push if still applicable: F5 (avoid the partially-initialised `JobMaster` via `CALLS_REAL_METHODS` plus reflective private-field injection), F6 (use imports instead of fully-qualified `java.lang.reflect.*` / `org.mockito.Mockito.*` references), and F7 (consolidate `clearPipelineMetricsForRestore` and `removeMetricsContext` into one helper with a log-level or reason parameter).
Once a new commit or description update lands covering items 1–3, I'll take another pass right away.
<!-- streview-comment:1799 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Reset metrics after pipeline recovery [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11495:
URL: https://github.com/apache/seatunnel/pull/11495#issuecomment-6258969964
Thanks for the update on 9c047310570519504bd5d37be03421b46b377dba. Before I can mark the earlier points resolved I'd like to confirm them against the diff itself rather than the review summary, so a few pointers would help:
- **Stale task-group contexts (F1)** – could you point me to where the restore path now instructs the surviving workers to drop the failed attempt's task-group contexts, so they stop re-reporting them?
- **Clear vs. periodic report race (F2)** – as I read it the stale-report window still exists and is documented rather than closed. Documenting the race doesn't resolve a HIGH bug on its own, so can you explain how the retry is prevented from being seeded with prior-attempt counters if a report lands between the clear and the new deploy, or why that window is acceptable? On placement relative to `reset()`: both locations behave the same today, and I'm fine with it staying next to the other restore-path steps.
- **Counters restarting from zero (F3)** – please point me to the incompatible-change / docs note so I can verify the wording.
- **Tests (F4)** – which tests cover the cleanup-before-removal order, scope isolation, and the cleanup-failure path? Do any exercise the real metrics store rather than only verifying a mock invocation?
The low-severity style notes (F5–F7) are non-blocking and can go to a follow-up if you prefer. Once the above is confirmed I'm happy to approve.
<!-- streview-comment:1873 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Reset metrics after pipeline recovery [seatunnel]
Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang commented on PR #11495:
URL: https://github.com/apache/seatunnel/pull/11495#issuecomment-6339127822
Dev sync done: #12938 ([Fix][Transform-V2] Restore archived LLM wrapper deserialization) has landed on `dev`, and the merge (`66f46aa37`) is pushed. Verified locally on the merged tree: `LLMBooleanOutputTest` now passes 42/42 and `StateTransitionCleanupTest` passes 13/13, so the inherited unit-test failure from the previous Build is resolved. The new Build on this head should be clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Reset metrics after pipeline recovery [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11495:
URL: https://github.com/apache/seatunnel/pull/11495#issuecomment-6338491368
Thanks @SEZ9, @zhangshenghang and @nzw921rx. I checked the disputed restore path on `68f3b7c5`:
`SubPlan.prepareRestorePipeline` calls `jobMaster.clearPipelineMetricsForRestore` at 497, before reset. That method calls `cleanTaskGroupContext` at `JobMaster.java:1289` before removing metrics at 1294. The helper sends `CleanTaskGroupContextOperation` to surviving workers at 1321-1325. So the statement that this restore path never sends the operation does not match this head.
The remaining in-flight-report race is real and explicitly documented in the method Javadoc (1280-1283): this is best-effort cleanup, not an attempt-fenced metric store or proof that every observable snapshot is zero. I retained that limitation in my previous review rather than claiming exact fencing. EN incompatible-changes.md lines 8-13 already describes the streaming-counter reset and dashboard implications; the ZH entry mirrors it. The shared helper also exists, so its use should be checked before requesting another consolidation. My previous focused test-coverage and cleanup-error-handling suggestions remain non-blocking; I am not closing your review by proxy.
There is also a CI correction: the run has completed with failure, superseding my queued note. Besides the four LLM-unit failures, the current run lists engine-v2, HBase, MySQL-CDC, transform-part-2 and all-connectors-part-2/6 failures. They cannot all be described simply as one backpressure error. The inspected logs include the LLM InvalidClassException and empty-template-provider failures in FlinkMetricsIT/TestSparkDateTimeTransformIT. Proposed fixes #12938 and #12895 are still open; the backpressure proposals #12313/#12316 are also open. Syncing current dev alone is not a verified cure. Please keep the existing source discussion separate from those failed stages and rerun affected lanes after their shared corrections are available. My own source conclusion remains unchanged; successful required CI and final maintainer review remain gates.
Current status evidence: head `68f3b7c5b75b17ba3c47bd9eac0c49b3f66fd7c2`, base `dev`, compare `diverged` (ahead 9, behind 8), mergeable state `blocked`.
CI references: [current Build check](https://github.com/apache/seatunnel/runs/119203277609), [Run / unit-test (11, ubuntu-latest)](https://github.com/zhangshenghang/seatunnel/actions/runs/39575592933/job/119141357504), [Run / all-connectors-it-6 (11, ubuntu-latest)](https://github.com/zhangshenghang/seatunnel/actions/runs/39575592933/job/119141358740).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Reset metrics after pipeline recovery [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11495:
URL: https://github.com/apache/seatunnel/pull/11495#issuecomment-6357400513
Thanks for the dev sync, @zhangshenghang. Since `66f46aa37` is only a merge of `dev` and the PR-owned hunks are identical to what I reviewed at `68f3b7c5b7`, my previous verdict stands and I'm not re-opening the review. For completeness, here is where the earlier points are on this head:
- **F1 (worker-side stale contexts)** – resolved: `clearPipelineMetricsForRestore` cleans the task-group contexts on the surviving workers before removing the pipeline's snapshots.
- **F2 (race with the periodic metrics report)** – accepted as a documented best-effort fix. An attempt-fenced store can be a follow-up.
- **F3 (semantics change for streaming jobs)** – resolved via the incompatible-changes note (en/zh) and the REST job-lifecycle docs.
- **F4–F7 (test coverage, test scaffolding, fully-qualified names, helper consolidation)** – unchanged and all non-blocking. If you'd like to pick up any of them in this PR, great; otherwise a short note that they're deferred is enough, and F7 may already be moot given the current two-step shape of `clearPipelineMetricsForRestore`, so feel free to just confirm.
Good to hear `StateTransitionCleanupTest` passes 13/13 on the merged tree. Nothing further from my side.
<!-- streview-comment:1975 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Reset metrics after pipeline recovery [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11495:
URL: https://github.com/apache/seatunnel/pull/11495#issuecomment-6135261008
Quick CI follow-up for head `0ac0f2e5a0`, since my last review said the fork run was still finishing.
It has now completed (`zhangshenghang/seatunnel` run 38112486866) and the only failures are `elasticsearch-connector-it (11)` and `(17)`, both in the `elasticsearch_auth_apikey_encoded_temp.conf` Flink container runs. That is outside this PR's three files (pipeline metrics clearing in `SubPlan` / `JobMaster`), and the same Elasticsearch IT failed on an unrelated PR in the same window. The test was stabilised on `dev` after this run by #12711 (polling for index visibility). The `unit-test (11)` and `engine-k8s-it` failures I listed earlier are no longer in the failed set, so please disregard them. This is an inference from the failing job names and logs.
So CI is not the blocker here. The code-side items in my last review (Issue 1, the failed attempt's task-group contexts are not cleaned before the metrics clear, so the store can be re-populated) and the similar points from @SEZ9 and @nzw921rx still apply to this head. Once you push a fix, I will re-review the latest head. When you do, please also sync with `dev` and rerun only the failed jobs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Reset metrics after pipeline recovery [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11495:
URL: https://github.com/apache/seatunnel/pull/11495#issuecomment-6266721701
Thanks @SEZ9. To save you a trip through the diff, here are the pointers on head `9c047310570519504bd5d37be03421b46b377dba` (read from source only; I have not run a failover reproduction):
- **F1 (stale task-group contexts)**: `SubPlan.prepareRestorePipeline` calls `jobMaster.clearPipelineMetricsForRestore(getPipelineLocation())` at `SubPlan.java:497`. In `JobMaster.java`, `clearPipelineMetricsForRestore` (line 1287) first calls `cleanTaskGroupContext(pipelineLocation)` (line 1289, defined at 1310), which sends `CleanTaskGroupContextOperation` to the still-alive owners, and only then calls `seaTunnelServer.removeMetrics(pipelineLocation)` (line 1303).
- **F2 (race)**: it is documented, not closed. The window is an in-flight report, plus the case where the clean fails on a worker that is still alive. I asked @zhangshenghang above to state that case explicitly in the PR description so the PR does not over-claim a full fix for #11472. I agree with you that this is best-effort by design.
- **F3 (counters restart from zero)**: the note is in `docs/en/introduction/concepts/incompatible-changes.md` and the `docs/zh` twin, headed "Behavior change: runtime counters no longer accumulate values from a failed pipeline attempt", with a shorter note in `docs/{en,zh}/engines/zeta/rest-api-job-lifecycle.md`.
- **F4 (tests)**: in `StateTransitionCleanupTest`, `testRestoreClearRunsTaskGroupCleanupBeforeMetricsRemoval` covers the order, `testRestoreClearRemovesMetricsFromTheStateStore` uses the real `MetricsSnapshotStateStore` and checks other pipelines are untouched, and `testMetricsCleanupFailureDoesNotPreventRestore` covers the failure path. No test simulates a late report from the old attempt, so that case is not covered.
One new item for @zhangshenghang: GitHub currently reports this PR as `DIRTY` (merge conflicts with `dev`; the branch is `ahead_by=8, behind_by=49`). Please sync the latest `dev` and resolve the conflicts. `Build` is green on the current head, but it should be re-confirmed after the sync.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Reset metrics after pipeline recovery [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11495:
URL: https://github.com/apache/seatunnel/pull/11495#issuecomment-6183582630
Looked at the follow-up on `9c047310` against the earlier `f45b6de63a` state, focusing only on the points from the previous round.
**F1 / F2 (restore path and race with the worker report)** – The new ordering in `clearPipelineMetricsForRestore` looks like the right shape: clean the failed attempt's task-group contexts on the surviving workers via `CleanTaskGroupContextOperation` first, then drop the pipeline snapshots from the master store, all before `reset()` under `restoreLock`. Making `cleanTaskGroupContext` best-effort per task group (log and continue, re-set the interrupt flag on `InterruptedException`) instead of aborting the whole loop on the first failure is also an improvement, since one unreachable worker no longer leaves the other workers' stale contexts in place. One thing I'd like confirmed in the PR description: in the case where the clean operation fails for a worker that is still a member, what happens to that worker's `finishedExecutionContexts` — is it still possible for it to re-report the old attempt's counters after the snapshot has been removed? If so, please state that limitation e
xplicitly so we don't claim a complete fix for #11472.
**F3 (semantics change for streaming jobs)** – I see docs were touched in this revision. Please point me to the exact place where the "counters reset on pipeline retry" behaviour is described, and whether it is listed as an incompatible/behaviour change rather than only an inline note.
**F4 / F5 (tests)** – Please confirm whether `StateTransitionCleanupTest` still only verifies that the mock is invoked, or whether a test now exercises the actual restore path (real metrics store, worker re-report, new attempt starting from zero). If it is still mock-only, an e2e or integration-style test covering the failover scenario from #11472 would be the strongest evidence that the race is actually closed. The `CALLS_REAL_METHODS` + reflective private-field injection setup is still something I'd prefer to see simplified if the test is kept.
**F6 / F7 (style)** – Minor, but please confirm the fully-qualified `java.lang.reflect.*` / `org.mockito.Mockito.*` references were replaced with imports, and whether `clearPipelineMetricsForRestore` and `removeMetricsContext` have been consolidated into a single helper (or explain why they need to stay separate).
Once the `Build` run finishes and the above are confirmed, I'm happy to do a final pass.
<!-- streview-comment:1769 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Reset metrics after pipeline recovery [seatunnel]
Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx commented on code in PR #11495:
URL: https://github.com/apache/seatunnel/pull/11495#discussion_r4250001055
##########
seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/dag/physical/SubPlan.java:
##########
@@ -494,6 +494,7 @@ private boolean prepareRestorePipeline() {
String.format(
"Restore time %s, pipeline %s",
pipelineRestoreNum + "", pipelineFullName));
+ jobMaster.clearPipelineMetricsForRestore(getPipelineLocation());
Review Comment:
Is it more reasonable to put it in `reset()`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Reset metrics after pipeline recovery [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo merged PR #11495:
URL: https://github.com/apache/seatunnel/pull/11495
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Reset metrics after pipeline recovery [seatunnel]
Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx commented on code in PR #11495:
URL: https://github.com/apache/seatunnel/pull/11495#discussion_r3631704158
##########
seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/dag/physical/StateTransitionCleanupTest.java:
##########
@@ -97,6 +105,57 @@ void testPhysicalPlanIgnoresLateTransitionWhenJobStateAlreadyTerminal() throws E
Assertions.assertEquals(JobStatus.FAILED, planWithStateMaps.runningJobState.get(jobId));
}
+ @Test
+ void testPipelineRestoreClearsMetricsFromTheFailedAttempt() throws Exception {
+ long jobId = instance.getFlakeIdGenerator(Constant.SEATUNNEL_ID_GENERATOR_NAME).newId();
+ JobImmutableInformation jobImmutableInformation = mock(JobImmutableInformation.class);
+ JobConfig jobConfig = mock(JobConfig.class);
+ Map<String, Object> envOptions = new HashMap<>();
+ envOptions.put("job.retry.times", "1");
+ envOptions.put("job.retry.interval.seconds", "0");
+ when(jobImmutableInformation.getJobId()).thenReturn(jobId);
+ when(jobImmutableInformation.getJobConfig()).thenReturn(jobConfig);
+ when(jobConfig.getEnvOptions()).thenReturn(envOptions);
+
+ IMap<Object, Object> runningJobState =
+ nodeEngine.getHazelcastInstance().getMap("restoreMetricsState-" + jobId);
+ IMap<Object, Long[]> runningJobStateTimestamps =
+ nodeEngine.getHazelcastInstance().getMap("restoreMetricsStateTimestamps-" + jobId);
+ PipelineLocation pipelineLocation = new PipelineLocation(jobId, 1);
+ runningJobState.put(pipelineLocation, PipelineStatus.FAILED);
+
+ ExecutorService executorService = Executors.newSingleThreadExecutor();
+ try {
+ SubPlan subPlan =
+ new SubPlan(
+ 1,
+ 1,
+ System.currentTimeMillis(),
+ Collections.emptyList(),
+ Collections.emptyList(),
+ jobImmutableInformation,
+ executorService,
+ runningJobState,
+ runningJobStateTimestamps,
+ Collections.emptyMap());
+ JobMaster jobMaster = mock(JobMaster.class);
+ CheckpointManager checkpointManager = mock(CheckpointManager.class);
+ PhysicalPlan physicalPlan = mock(PhysicalPlan.class);
+ when(jobMaster.getCheckpointManager()).thenReturn(checkpointManager);
+ when(jobMaster.getPhysicalPlan()).thenReturn(physicalPlan);
+ subPlan.setJobMaster(jobMaster);
+
+ java.lang.reflect.Method restore =
Review Comment:
please use `import java.lang.reflect.Method `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Reset metrics after pipeline recovery [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11495:
URL: https://github.com/apache/seatunnel/pull/11495#issuecomment-6190329968
CI correction for head `9c047310570519504bd5d37be03421b46b377dba`: in my last review I said the required `Build` run was still in progress. That has changed. The apache-side `Build` check on this head now shows `success`, so the technical state from my side is: no source-level blocker and CI green; only the maintainer approval gate remains.
Regarding @SEZ9's latest questions (the limitation statement about a worker whose clean operation fails, where the "counters reset on retry" behavior is documented, and whether any test goes beyond mock invocation): those are directed at the author. I agree they are worth a short answer in the PR description, in particular stating plainly whether a still-alive worker whose `CleanTaskGroupContextOperation` fails could re-report the old attempt's counters, so the PR does not over-claim a complete fix for #11472. I will re-check once a new commit or description update lands.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Reset metrics after pipeline recovery [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on code in PR #11495:
URL: https://github.com/apache/seatunnel/pull/11495#discussion_r4291809649
##########
seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/dag/physical/SubPlan.java:
##########
@@ -494,6 +494,7 @@ private boolean prepareRestorePipeline() {
String.format(
"Restore time %s, pipeline %s",
pipelineRestoreNum + "", pipelineFullName));
+ jobMaster.clearPipelineMetricsForRestore(getPipelineLocation());
Review Comment:
Thanks @nzw921rx, good question. `reset()` is private and `prepareRestorePipeline` is its only caller, so the two placements behave the same today. I would keep the call next to the other restore-path steps where it is, because `reset()` resets the in-memory SubPlan state and the clear talks to workers and the master metrics store, but I see it as a style choice and @zhangshenghang can decide.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Reset metrics after pipeline recovery [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11495:
URL: https://github.com/apache/seatunnel/pull/11495#issuecomment-6137429813
Thanks @zhangshenghang. The ordering you describe for `f45b6de6` (`cleanTaskGroupContext(pipelineLocation)` before `seaTunnelServer.removeMetrics(...)` in `clearPipelineMetricsForRestore`) is the right shape for Issue 1, but I can't yet see the diff for that head, so I'm treating it as unverified until I can review the pushed change. The remaining items from my previous review are:
1. **Task-group context cleanup before the metrics clear (Issue 1).** To be confirmed against the `f45b6de6` diff, including that a cleanup failure is logged and never blocks the restore as you describe.
2. **Ordering vs. the periodic metrics report.** Even with the contexts cleaned first, a worker report snapshotted before the clean could still land after the clear. Could you explain how that window is handled (e.g. the clear runs only after the clean is acknowledged, or reports for an unregistered task group are ignored)? If it is still probabilistic, let's note that explicitly and decide whether it's acceptable.
3. **Semantics / docs note.** Counters that previously survived a pipeline retry via the IMap snapshot now reset to zero on every restore. Please add a short note to the Zeta metrics docs or the incompatible-changes list so streaming users aren't surprised.
4. **Test coverage.** Glad `StateTransitionCleanupTest` is green locally. The concern remains that the tests only verify a mocked method is invoked. It would be good to have at least one test that asserts `cleanTaskGroupContext` runs before `removeMetrics` (in-order verification) and one that exercises a real metrics store / re-report path and asserts the restored attempt actually starts from zero. If the latter is too heavy for a unit test, an engine IT is fine.
5. **Test scaffolding and style (minor).** The partially-initialised `JobMaster` via `CALLS_REAL_METHODS` plus reflective private-field injection is brittle; constructing the needed pieces directly or extracting the clear/clean logic into a small testable unit would help. Please also replace the fully-qualified `java.lang.reflect.*` / `org.mockito.Mockito.*` references with imports.
6. **Helper duplication (minor).** `clearPipelineMetricsForRestore` looks like a copy of `removeMetricsContext` with a different log level; please consolidate into one helper now that the restore path has an extra step.
I'll re-review as soon as the `f45b6de6` changes are visible here.
<!-- streview-comment:1715 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Reset metrics after pipeline recovery [seatunnel]
Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang commented on PR #11495:
URL: https://github.com/apache/seatunnel/pull/11495#issuecomment-6359424711
Update on the `Build`: the failed-jobs rerun (attempt 2) reproduced the **exact same failure set** as attempt 1 — the same 10 IT jobs (`all-connectors-it-2/6`, `transform-v2-it-part-2`, `mysql-cdc-connector-it`, `hbase-connector-it`, both JDK matrices) with the same first error (`TestSparkDateTimeTransformIT`: `PreconditionViolationException` — Testcontainers invocation context provider returned an empty stream, 68/69 tests in the suite pass). An identical failure set across two independent runs points to a systematic runner-environment issue for these suites on the fork, not to this PR's changes (a Zeta metrics fix) and not to random flake. `unit-test` passes on both attempts.
I have therefore stopped rerunning (two identical attempts is enough evidence); the remaining red checks are environmental. The code itself carries two approvals (nzw921rx) and SEZ9's re-confirmation on this head.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Reset metrics after pipeline recovery [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11495:
URL: https://github.com/apache/seatunnel/pull/11495#issuecomment-6331445499
Thanks both.
@zhangshenghang — thanks for the breakdown of the `Build` failures. Agreed that the `LLMBooleanOutputTest.readsLegacyFactoryWrapperWithValidationDisabled` failure (`serialVersionUID = 7090258061839605137 vs -3241928148724104003`) is inherited: `AbstractMultiCatalogTransform` isn't touched here, and dev's own run at `bab2d089c` ([run 39536327852](https://github.com/apache/seatunnel/actions/runs/39536327852)) shows the same exception. Syncing `dev` into this branch once the upstream fix lands sounds right; no rerun needed in the meantime. The `BackpressureSlowSinkIT` error I'm also happy to treat as the known flaky one.
@nzw921rx — thanks for the review.
That said, the CI picture doesn't change the open items from my earlier review, which haven't been addressed in the thread yet. Before I can approve, I'd like to see the following resolved or explicitly discussed:
- **F1 / F2 (HIGH)**: the restore path still never sends `CleanTaskGroupContextOperation`, so surviving workers keep the failed attempt's `TaskGroupContext`s in `finishedExecutionContexts` and re-report them on the next metrics backup interval; and the IMap clear in `SubPlan.java` races with that periodic report, so stale snapshots can be re-merged before the new attempt deploys. As it stands the fix is probabilistic. Please either send the clean operation to the workers as part of restore, or otherwise explain how the re-report is prevented.
- **F3 (MEDIUM)**: the unconditional reset changes metrics semantics for streaming jobs (counters previously carried across restarts now drop to zero on every pipeline retry). Please add a doc / incompatible-changes note, or make the behaviour opt-in.
- **F4 (MEDIUM)**: `StateTransitionCleanupTest` currently only verifies that a mock method is invoked. Please add coverage that goes through the real metrics store and the worker re-report path and asserts the restored attempt actually starts from zero.
- **F5 / F6 / F7 (LOW)**: the `CALLS_REAL_METHODS` + reflective private-field injection scaffolding, the fully-qualified `java.lang.reflect.*` / `org.mockito.Mockito.*` references, and the duplicated body between `clearPipelineMetricsForRestore` and `removeMetricsContext` (consolidate into one helper).
If any of these have already been handled in a push I missed, a short pointer in the thread is enough and I'll re-review.
<!-- streview-comment:1935 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org