You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "agarwalrahul2702 (via GitHub)" <gi...@apache.org> on 2026/07/18 22:47:42 UTC

[PR] [Fix][Zeta] Fix stop-with-savepoint hang in DOING_SAVEPOINT [seatunnel]

agarwalrahul2702 opened a new pull request, #11489:
URL: https://github.com/apache/seatunnel/pull/11489

   # [Fix][Zeta] Fix stop-with-savepoint hang in DOING_SAVEPOINT
   
   Fixes #11473
   
   ### Root cause
   
   Two defects combined to produce the reported hang:
   
   1. **Savepoint barriers could never be injected while the source was emitting a large split.**
      `SourceFlowLifeCycle.triggerBarrier` must acquire the checkpoint lock to inject a
      checkpoint/savepoint barrier into the source. `FakeSourceReader.pollNext` emitted the
      **entire split** in a single call while holding that same lock. With the issue's config
      (`row.num = 100000000`, `split.num = 1`) the lock is held for the whole 100M-row split —
      effectively forever. A thread dump taken during the hang shows the barrier thread
      `BLOCKED` on the checkpoint lock whose owner is the source task thread busy inside
      `FakeDataGenerator` (the sink meanwhile idle-polling an empty queue — this is not
      backpressure). Consequently every checkpoint and savepoint expires at
      `checkpoint.timeout`, which also explains `SinkCommittedCount = 0` in the issue report.
   
   2. **A failed savepoint left the job wedged in `DOING_SAVEPOINT` forever.**
      `JobMaster.savePoint()` sets `DOING_SAVEPOINT`, triggers savepoint barriers, and waits.
      When the savepoint checkpoint expires, the checkpoint error handling restores the
      pipeline and the job keeps running — but nothing ever reverted the job status. The job
      stayed in `DOING_SAVEPOINT` indefinitely with `errorMsg = null` (exactly the reported
      symptom), while `/finished-jobs/SAVEPOINT_DONE` never listed it and slots stayed
      occupied until a manual normal stop.
   
   ### Solution
   
   - `FakeSourceReader` now emits at most 4096 rows per `pollNext` call and requeues the
     remainder of the split at the head of the split deque, releasing the checkpoint lock
     between batches so barriers can be injected. The requeued split carries the remaining
     row count, so a snapshot taken between batches captures the not-yet-emitted rows
     (state semantics preserved for restore). Split-read-interval pacing and the idle sleep
     are preserved at split granularity; user-configured custom rows (`rows` option) are
     emitted in one batch as before.
   - `JobMaster.savePoint()` now reverts the job state from `DOING_SAVEPOINT` back to
     `RUNNING` (new `PhysicalPlan#savepointFailed`) when the savepoint does not complete, so
     the failure is surfaced to the caller instead of wedging the job. No timeout or forced
     cancellation is introduced; the savepoint completion path itself is unchanged.
   
   ### Reproduction (before the fix, on current dev)
   
   In-process Zeta cluster (`AbstractSeaTunnelServerTest`) running the issue's exact
   FakeSource → Console streaming config:
   
   - Job reaches `RUNNING`, savepoint requested → status turns `DOING_SAVEPOINT`, the
     savepoint pending checkpoint is never created, the in-flight checkpoint expires after
     `checkpoint.timeout`, the pipeline restores, and the job stays `DOING_SAVEPOINT`
     indefinitely; the stop request eventually fails with
     `CheckpointException: CheckpointCoordinator shutdown` while the status remains wedged.
   - A baseline run of the same config **without** any savepoint shows every periodic
     checkpoint expiring too (barrier starvation is independent of the savepoint request).
   - A normal stop (`isStopWithSavePoint=false`) cleans the job up, as reported.
   
   ### Regression test
   
   `SavePointBusySourceTest` (engine-server) proves, against a real in-process Zeta cluster:
   
   1. stop-with-savepoint **completes** under a busy source emitting a huge split;
   2. the job passes `DOING_SAVEPOINT` → `SAVEPOINT_DONE`;
   3. the stop request (the same `CoordinatorService#savePoint` call the REST
      `/stop-job` handler uses) returns instead of hanging;
   4. all slots are released (`SlotService` worker profile shows 0 assigned slots);
   5. a normal stop while the source is busy still works (`CANCELED`, slots released);
   6. a savepoint that cannot complete (sink delays `prepareCommit` past
      `checkpoint.timeout`) reports an `ExecutionException` to the caller and the job
      reverts to `RUNNING` instead of sticking in `DOING_SAVEPOINT`, then reaches a
      terminal state on stop with slots released.
   
   ### Verification (local, JDK 11 / Temurin 11.0.31, macOS arm64)
   
   | Command | Result |
   |---|---|
   | `./mvnw -pl seatunnel-engine/seatunnel-engine-server -Dtest=SavePointBusySourceTest test` | Tests run: 3, Failures: 0, Errors: 0 |
   | `./mvnw -pl seatunnel-engine/seatunnel-engine-server -Dtest=SavePointTest test` | Tests run: 6, Failures: 0, Errors: 0, Skipped: 1 (pre-existing `@Disabled`) |
   | `./mvnw -pl seatunnel-connectors-v2/connector-fake test` | Tests run: 12, Failures: 0, Errors: 0 |
   | `./mvnw -pl seatunnel-engine/seatunnel-engine-server -Dtest=CheckpointTimeOutTest,CheckpointBarrierTriggerErrorTest,CheckpointPlanTest,CheckpointErrorRestoreEndTest,CheckpointManagerTest,CheckpointCoordinatorTest,CheckpointSerializeTest,JobMasterTest test` | Tests run: 24, Failures: 0, Errors: 0, Skipped: 2 (pre-existing) |
   | `./mvnw spotless:apply` (changed modules) | BUILD SUCCESS (formatting applied) |
   | `./mvnw -DskipTests verify` (full repo, 284 modules) | BUILD SUCCESS (55:46 min) |
   
   ### Checklist
   
   - [x] Code changed follows the project code style (spotless applied)
   - [x] New files carry the ASF license header
   - [x] Regression test added
   - [x] No user-facing config options changed; no docs impact
   - [x] No new dependencies
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Zeta] Fix stop-with-savepoint hang in DOING_SAVEPOINT [seatunnel]

Posted by "agarwalrahul2702 (via GitHub)" <gi...@apache.org>.
agarwalrahul2702 commented on PR #11489:
URL: https://github.com/apache/seatunnel/pull/11489#issuecomment-5254435089

   Pushed commit `438e62754b` to address the red `unit-test (11, ubuntu-latest)` failure from the latest run.\n\nRoot cause: the partial-savepoint-failure fixture configured the supposed successful pipeline with `split.read-interval = 3000` while `checkpoint.timeout = 2000`, so CI could legitimately time out both pipelines and fail the assertion that at least one coordinator reached `SUSPEND`. The fixture now uses `split.read-interval = 100` for the successful pipeline, while keeping the `InMemory` sink timeout path unchanged for the failing pipeline.\n\nValidation run locally:\n- `git diff --check`\n- `./mvnw -pl seatunnel-ci-tools -Dtest=ImportClassCheckTest#javaUtilCompletableFutureCheck test`\n\nThe other red jobs from the previous run were connector E2E failures outside this PR's touched code: `DatabendCDCSinkIT` timed out with 0 rows, and `ElasticsearchAuthIT` failed creating an API key because Elasticsearch returned 503 cluster-state-not-recovered. Those should be re-evaluated
  on the new CI run.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Zeta] Fix stop-with-savepoint hang in DOING_SAVEPOINT [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11489:
URL: https://github.com/apache/seatunnel/pull/11489#issuecomment-5379386592

   Thanks for the +1, @dybyte!
   
   Quick status note since your approval was conditional on CI: as of right now, the `Build` check on the current head `2b840e97663480e10bef169ac963a0ae6580dccb` is still `failure` (https://github.com/apache/seatunnel/runs/96156779577). No new commit has landed since my two APPROVED reviews on this head, so nothing has changed on my side — this is just a live-status update.
   
   I'm not re-deriving the failing jobs from scratch in this note; my permission on this repository is read-only, so I can't trigger a rerun myself. The concrete next step is for @agarwalrahul2702 to rerun the failed jobs on the fork's `Build` run and confirm it goes green.
   
   One more thing worth flagging so nobody is confused by the review summary: `reviewDecision` on this PR is still showing `CHANGES_REQUESTED` overall, but that's aggregating the earlier-round CHANGES_REQUESTED states from @nzw921rx, @davidzollo, and an early round from @SEZ9 that haven't been re-submitted as approvals against this head — it's not a new blocker from any of them on the current diff as far as I can tell. My own APPROVED conclusion isn't conditional on anything further from the source side.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Zeta] Fix stop-with-savepoint hang in DOING_SAVEPOINT [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11489:
URL: https://github.com/apache/seatunnel/pull/11489#issuecomment-5332090037

   结论:当前头暂不可交付。
   
   源码正确性:重试用例在任务刚进入 RUNNING 后就直接伪造 checkpoint coordinator 已就绪,随后第一次请求被拒绝、第二次请求在真实 task-ready 握手尚未完成时启动,最终在 60 秒 checkpoint timeout 后将作业置为 FAILED。应先等待真实 coordinator ready,再注入一次 false 的前置条件失败,之后恢复 ready 并重试。最小补丁已推送到维护者 fork:<https://github.com/DanielLeens/seatunnel/commit/fb8618fe928a700cb4611f6618ee0a75e6dd7a44>。
   
   CI 归因:当前头 8f5d15cd 的 unit-test 作业 95686043072 确认失败于 SavePointBusySourceTest.testSavepointStartPreconditionFailureCanBeRetriedAfterReady;断言等待 SAVEPOINT_DONE,实际为 FAILED,并非外部基础设施失败。
   
   范围风险:当前头同时加入了 TaskExecutionService 的 cooperative task close 生产逻辑,但该路径不由上述 savepoint precondition rejection 触发,且没有针对该新行为的回归测试。请将其拆分为独立 PR,或补充与本问题的因果关系和完整测试证明。
   
   审查与合并门禁:未发现未解决的 review thread,但 reviewDecision 仍为 CHANGES_REQUESTED,mergeStateStatus 为 BLOCKED。尽管 PR 标示 maintainer 可修改,GitHub 拒绝 DanielLeens 直接推送到作者分支(403),因此请作者 cherry-pick 上述提交并移除或拆分无关生产改动;新当前头 CI 通过后再复核。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Zeta] Fix stop-with-savepoint hang in DOING_SAVEPOINT [seatunnel]

Posted by "agarwalrahul2702 (via GitHub)" <gi...@apache.org>.
agarwalrahul2702 commented on PR #11489:
URL: https://github.com/apache/seatunnel/pull/11489#issuecomment-5317852719

   Pushed `93a7a6672` to address the latest SEZ9 review on head `a33faca4d`.
   
   What changed:
   - `JobMaster` now preserves every per-pipeline savepoint exception instead of classifying the whole job from the first exception in pipeline-list order.
   - The restore-to-`RUNNING` path is now allowed only when no pipeline reached `SUSPEND` and every collected failure is a pre-start savepoint rejection.
   - `CHECKPOINT_COORDINATOR_SHUTDOWN` is handled alongside `TASK_NOT_ALL_READY_WHEN_SAVEPOINT` as a retryable pre-start rejection.
   - Any mixed result containing a genuine checkpoint failure, any partial suspension, interruption, or non-precondition failure still routes to the deterministic stop fallback.
   - Added comments/Javadoc around the aggregation/classification rule so the lifecycle distinction is explicit.
   
   Coverage added:
   - `JobMasterTest#testSavepointPreconditionClassificationRequiresAllFailuresPreconditionOnly` covers the mixed multi-pipeline case: one precondition failure plus one genuine `CHECKPOINT_EXPIRED` failure must not be restored to `RUNNING`.
   - `JobMasterTest#testSavepointPreconditionClassificationAcceptsAllPreStartFailures` covers all-pre-start failures, including `CHECKPOINT_COORDINATOR_SHUTDOWN`.
   - `SavePointBusySourceTest#testSavepointStartPreconditionFailureKeepsJobRunning` now also flips readiness back to true and verifies a retry reaches `SAVEPOINT_DONE` on the same job.
   
   Local validation:
   - `./mvnw -pl seatunnel-engine/seatunnel-engine-server spotless:apply` passes.
   - `git diff --check` passes.
   - `./mvnw -pl seatunnel-ci-tools -Dtest=ImportClassCheckTest#javaUtilCompletableFutureCheck test` passes.
   
   I also re-attempted the targeted engine-server tests locally, but this cleaned workspace still cannot compile `seatunnel-engine-server` because Maven resolves mismatched/stale local SNAPSHOT artifacts before test execution (`RestoreMode`, API metadata/signal classes, metric names, and generated Lombok members are missing). The PR CI has been triggered on the new head and should be the authoritative full-reactor verification.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Zeta] Fix stop-with-savepoint hang in DOING_SAVEPOINT [seatunnel]

Posted by "agarwalrahul2702 (via GitHub)" <gi...@apache.org>.
agarwalrahul2702 commented on PR #11489:
URL: https://github.com/apache/seatunnel/pull/11489#issuecomment-5323268643

   Pushed `8e4364915` after syncing the branch with current `upstream/dev`.
   
   This addresses Daniel’s 2026-08-18 review:
   
   - Split the flaky retry assertion out of `testSavepointStartPreconditionFailureKeepsJobRunning`.
     - The busy-source reproduction now only verifies the pre-start savepoint failure keeps the job `RUNNING`, then cancels and checks slot cleanup.
     - It no longer tries to prove retry success against the deliberately checkpoint-hostile 100M-row / 60s-timeout config that failed in CI.
   - Added `testSavepointStartPreconditionFailureCanBeRetriedAfterReady` with a separate lightweight long-running FakeSource config (`stream_fakesource_retryable_to_console_savepoint.conf`) so retry success is verified without racing the original busy-source timeout budget.
   - Added the missing comment on `isSavepointStartPreconditionException()`.
   - Updated caller-facing failure selection so mixed failures prefer a non-precondition/genuine failure over a benign first precondition failure, and added `JobMasterTest#testSavepointFailureExceptionPrefersNonPreconditionFailure` for that behavior.
   
   Local validation:
   
   - `./mvnw -pl seatunnel-engine/seatunnel-engine-server spotless:apply` passes.
   - `git diff --check` passes.
   - `./mvnw -pl seatunnel-ci-tools -Dtest=ImportClassCheckTest#javaUtilCompletableFutureCheck test` passes.
   
   I attempted the targeted engine-server tests both directly and with `-am` after syncing `dev`:
   
   - Direct module run still fails before tests because Maven resolves stale/mismatched SNAPSHOT artifacts for reactor modules.
   - Reactor `-am -DfailIfNoTests=false` gets past the no-tests module but still stops before engine-server in `seatunnel-config-shade` because the generated/shaded Typesafe Config classes are missing in this local workspace.
   
   So the PR CI run for head `8e4364915` is the authoritative full-reactor verification for the affected engine tests.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Zeta] Fix stop-with-savepoint hang in DOING_SAVEPOINT [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11489:
URL: https://github.com/apache/seatunnel/pull/11489#issuecomment-5381238607

   Thanks @DanielLeens for the thorough from-scratch re-review on `2b840e97663480e10bef169ac963a0ae6580dccb` — great to see both of your previous blockers confirmed resolved, with the `TaskExecutionService.java` change removed entirely and the test TOCTOU risk closed via the added `awaitCheckpointCoordinatorsReady(jobMaster)` calls in `SavePointBusySourceTest.java`. Your confirmation that the rest of the diff is byte-for-byte unchanged since your last round is much appreciated.
   
   Thanks also @dybyte for the +1.
   
   Remaining asks before merge:
   
   1. Per @DanielLeens's process note, the overall `reviewDecision` is still `CHANGES_REQUESTED` due to stale reviews from earlier rounds. I'll ask those reviewers to re-affirm against the current head, or a maintainer with write access can dismiss the stale reviews so this clears the merge gate.
   2. Per @dybyte's condition, we need CI to pass on the current head before merging.
   
   Once those two are done, this should be good to go. Thanks all for the careful work on the stop-with-savepoint hang fix.
   
   <!-- streview-comment:451 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Zeta] Fix stop-with-savepoint hang in DOING_SAVEPOINT [seatunnel]

Posted by "agarwalrahul2702 (via GitHub)" <gi...@apache.org>.
agarwalrahul2702 commented on PR #11489:
URL: https://github.com/apache/seatunnel/pull/11489#issuecomment-5021838942

   Thanks for the review. I pushed `9e45d1415` to address the blocking API compatibility item.
   
   - Restored the existing public `CheckpointManager#triggerSavePoints()` signature to return `PassiveCompletableFuture<CompletedCheckpoint>[]`.
   - Added a separate additive method, `triggerSavePointsAndWaitComplete()`, for the `JobMaster` stop-with-savepoint path that needs finalized checkpoint-coordinator state.
   - Updated `JobMaster` to call the new method, so downstream callers of the existing public API keep the prior return type/behavior.
   
   For the low-severity config/docs note: this PR does not add or change any user-facing config option. The only config-like addition is a test resource used by the regression test, so I left user docs/examples unchanged.
   
   Verification after this change:
   
   - `./mvnw spotless:check` passes.
   - `./mvnw spotless:apply` passes.
   - `git diff --check` passes.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Zeta] Fix stop-with-savepoint hang in DOING_SAVEPOINT [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11489:
URL: https://github.com/apache/seatunnel/pull/11489#issuecomment-5192169627

   Thanks for your contribution! This is a solid addition to the project, and your attention to detail really shows. Contributors like you are what keep this community moving forward. Feel free to keep the PRs coming — we're always glad to have you here!
   
   Please ignore the following message if you have joined in the Slack~
   ```
   To facilitate focused communication among contributors, we’ve created a brand-new dedicated #contributors channel in official Slack workspace.
   Slack workspace link: https://s.apache.org/seatunnel-slack
   
   This channel is designed for technical discussions, feature design brainstorming, roadmap sync and direct communication with project core Contributors. Everyone who contributes code, docs, test cases or community ideas is warmly invited to join, share thoughts and collaborate together under The Apache Way. Welcome to join 😄
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Zeta] Fix stop-with-savepoint hang in DOING_SAVEPOINT [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo merged PR #11489:
URL: https://github.com/apache/seatunnel/pull/11489


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Zeta] Fix stop-with-savepoint hang in DOING_SAVEPOINT [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11489:
URL: https://github.com/apache/seatunnel/pull/11489#issuecomment-5366241903

   Thanks @DanielLeens for the thorough from-scratch re-review of `2b840e976`, and for confirming the retry-test ordering issue from your `037ea4335` review is now resolved.
   
   To recap on that single point: the problem was that the retry case faked the checkpoint coordinator as ready immediately after the task entered RUNNING, so the second savepoint request could start before the real task-ready handshake completed, eventually failing the job on the 60s checkpoint timeout. The fix in `2b840e976` follows exactly the approach you suggested — `SavePointBusySourceTest.java` now calls `awaitCheckpointCoordinatorsReady(jobMaster)` to wait for the real coordinator to be ready before each `setCheckpointCoordinatorsReady(jobMaster, false)` injection, so the precondition failure is injected deterministically rather than racing the handshake. Glad your verification confirms both `setCheckpointCoordinatorsReady(jobMaster, false)` call sites are now guarded and that nothing else moved between `037ea43357f03827622196d26f85e1bff2680568` and `2b840e97663480e10bef169ac963a0ae6580dccb` (`+2/-27`, with the `TaskExecutionService.java` change removed outright).
   
   Given both of your approvals on the current head, I have no remaining code asks on this point. Only two housekeeping items before merge: please confirm the APPROVED state stands on `2b840e976` as the final head, and flag here if you'd like the two-commit history squashed on merge or kept as-is. Otherwise I'll proceed once you confirm.
   
   Thanks again for the careful, byte-level re-verification — it made this round easy to close out.
   
   <!-- streview-comment:419 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Zeta] Fix stop-with-savepoint hang in DOING_SAVEPOINT [seatunnel]

Posted by "agarwalrahul2702 (via GitHub)" <gi...@apache.org>.
agarwalrahul2702 commented on PR #11489:
URL: https://github.com/apache/seatunnel/pull/11489#issuecomment-5323782273

   Pushed `ac936f844` for the latest CI/review feedback.
   
   What changed:
   - Removed the newly-added retry-success integration fixture/test that failed in CI by racing `SAVEPOINT_DONE` against the busy-source checkpoint timeout.
   - Deleted `stream_fakesource_retryable_to_console_savepoint.conf`, since that fixture existed only for the unstable retry assertion.
   - Kept `SavePointBusySourceTest.testSavepointStartPreconditionFailureKeepsJobRunning` focused on the deterministic requirement from #11473: an early savepoint precondition failure must fail the request but leave the job `RUNNING`, with cleanup verified by explicit cancel.
   - Kept the deterministic `JobMasterTest` coverage for multi-pipeline classification and mixed-failure reporting.
   - Synced the branch with current `upstream/dev` (`4a65e98ad`).
   
   This also addresses the two non-blocking notes from the latest review that were already present on the prior head:
   - `isSavepointStartPreconditionException()` has a dedicated comment explaining the pre-checkpoint boundary.
   - mixed failures prefer a non-precondition exception when surfacing the failure to the caller.
   
   Local validation on the pushed head:
   - `./mvnw -pl seatunnel-engine/seatunnel-engine-server spotless:apply` passes.
   - `git diff --check` passes.
   - `./mvnw -pl seatunnel-ci-tools -Dtest=ImportClassCheckTest#javaUtilCompletableFutureCheck test` passes.
   
   I also re-attempted `./mvnw -pl seatunnel-engine/seatunnel-engine-server -Dtest=JobMasterTest test`; this local workspace still fails during engine-server compile before tests because direct module resolution uses stale/missing intra-repo SNAPSHOT classes (`RestoreMode`, API metadata/signal/error classes, metric constants). The new fork Build is queued here: https://github.com/agarwalrahul2702/seatunnel/actions/runs/32100814768


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Zeta] Fix stop-with-savepoint hang in DOING_SAVEPOINT [seatunnel]

Posted by "agarwalrahul2702 (via GitHub)" <gi...@apache.org>.
agarwalrahul2702 commented on PR #11489:
URL: https://github.com/apache/seatunnel/pull/11489#issuecomment-5276022745

   Pushed `a33faca4d` to address the latest blocking review finding.
   
   Root cause addressed: `TASK_NOT_ALL_READY_WHEN_SAVEPOINT` means the savepoint was rejected before a checkpoint could start. The previous `finally` path treated that the same as a mid-flight savepoint failure and called `physicalPlan.savepointFailed()`, which force-stopped the job and made client retries impossible.
   
   Fix:
   - `JobMaster.savePoint()` now inspects the per-pipeline savepoint result before deciding the cleanup path.
   - If the failure is `CheckpointException(TASK_NOT_ALL_READY_WHEN_SAVEPOINT)` and no pipeline reached `SUSPEND`, it restores the global job state from `DOING_SAVEPOINT` back to `RUNNING` so the caller can retry.
   - If any pipeline already reached `SUSPEND`, or the failure is not the not-ready precondition, the existing deterministic `savepointFailed()` stop fallback remains in place to avoid reporting a partially stopped job as running.
   - Added `SavePointBusySourceTest.testSavepointStartPreconditionFailureKeepsJobRunning`, which forces the checkpoint coordinator readiness flag false, calls savepoint, asserts the request fails, and verifies the job remains `RUNNING` and still has a `JobMaster`.
   
   Local verification:
   - `./mvnw -pl seatunnel-engine/seatunnel-engine-server spotless:apply` passes.
   - `git diff --check` passes.
   - `./mvnw -pl seatunnel-ci-tools -Dtest=ImportClassCheckTest#javaUtilCompletableFutureCheck test` passes.
   
   I also attempted the targeted server regression locally after installing `seatunnel-shade`; the direct module run reaches compilation but fails because this workspace mixes current checkout code with old Apache snapshot jars, and the reactor `-am` compile still stops earlier in `seatunnel-config-shade` on generated/shaded Typesafe Config classes before engine-server is reached. The GitHub workflow is running on the pushed head now: https://github.com/agarwalrahul2702/seatunnel/actions/runs/31667542329


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Zeta] Fix stop-with-savepoint hang in DOING_SAVEPOINT [seatunnel]

Posted by "agarwalrahul2702 (via GitHub)" <gi...@apache.org>.
agarwalrahul2702 commented on PR #11489:
URL: https://github.com/apache/seatunnel/pull/11489#issuecomment-5256311191

   Pushed `8b56ffc60` after checking the latest failed run at head `0e3189a30`.
   
   Findings from `unit-test (11, ubuntu-latest)` job `93826638838`:
   - `SavePointBusySourceTest` now passes: `Tests run: 5, Failures: 0, Errors: 0`.
   - The remaining unit-test failure was `SeaTunnelClientTest.testSavePointAndRestoreWithSavePoint`.
   - Error: `SavePointFailedException: The job with id ... not running, save point failed`.
   
   Root cause: `SeaTunnelClientTest` uses `streaming_fake_to_console.conf`, but that fixture had FakeSource defaults (`row.num=5`, `split.num=1`, `split.read-interval=1ms`). On CI the job can finish between observing `RUNNING` and sending `savePointJob()`, so the master correctly reports that the job is no longer running.
   
   Fix in `8b56ffc60`:
   - Made the shared client streaming fixture actually long-running (`row.num=1000000`, `split.num=5`, `split.read-interval=100`). This also stabilizes the other client tests that expect this fixture to remain `RUNNING` until canceled.
   - Hardened the existing savepoint retry predicate to retry both "Task not all ready" and the transient "not running" savepoint race, with null-safe cause handling.
   
   Local validation:
   - `./mvnw -pl seatunnel-engine/seatunnel-engine-client spotless:apply` passes.
   - `git diff --check` passes.
   - `./mvnw -pl seatunnel-ci-tools -Dtest=ImportClassCheckTest#javaUtilCompletableFutureCheck test` passes.
   
   Targeted local client test execution is still blocked by local unresolved `seatunnel-shade` SNAPSHOT dependency; GitHub CI is queued on the new head: https://github.com/agarwalrahul2702/seatunnel/actions/runs/31515616274


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Zeta] Fix stop-with-savepoint hang in DOING_SAVEPOINT [seatunnel]

Posted by "agarwalrahul2702 (via GitHub)" <gi...@apache.org>.
agarwalrahul2702 commented on PR #11489:
URL: https://github.com/apache/seatunnel/pull/11489#issuecomment-5344460225

   Pushed `2b840e976` to address Daniel’s latest review.
   
   Changes:
   - Removed the cooperative-task-close production change from this PR. That takes `TaskExecutionService.java` back out of the net PR diff, so the unrelated lifecycle/test-coverage blocker is no longer part of this savepoint fix.
   - Added `awaitCheckpointCoordinatorsReady(jobMaster)` before forcing `isAllTaskReady=false` in the retry-precondition tests. The retry test now waits for the real coordinator-ready handshake first, injects exactly one precondition rejection, restores readiness, and then retries the savepoint.
   
   The retry fixture already has the periodic-checkpoint isolation from `037ea4335` (`checkpoint.interval = 600000`, `checkpoint.timeout = 60000`), so the test is driven by the explicit savepoint calls rather than an overlapping periodic checkpoint.
   
   Local validation:
   - `./mvnw -pl seatunnel-engine/seatunnel-engine-server spotless:apply` passes.
   - `git diff --check` passes.
   - `./mvnw -pl seatunnel-ci-tools -Dtest=ImportClassCheckTest#javaUtilCompletableFutureCheck test` passes.
   
   New Build run for this head: https://github.com/agarwalrahul2702/seatunnel/actions/runs/32271489557


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Zeta] Fix stop-with-savepoint hang in DOING_SAVEPOINT [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11489:
URL: https://github.com/apache/seatunnel/pull/11489#issuecomment-5368802685

   Confirmed on both points:
   
   1. **APPROVED stands on `2b840e97663480e10bef169ac963a0ae6580dccb`.** That is still the current head (I re-checked before writing this), no commit has landed since my second from-scratch re-review, and nothing in that review needs revisiting on this head. My approval is not conditional on anything further from the author's side.
   
   2. **Squash vs. keep-as-is is not mine to call.** Daniel's permission on this repository is read-only, so I don't have merge authority here regardless of preference. That decision belongs to whichever maintainer with write access performs the actual merge — happy to leave it to their judgment (or the project's usual convention) rather than expressing one myself.
   
   One process note for whoever merges: `reviewDecision` on the PR is still showing `CHANGES_REQUESTED` overall, not because of anything outstanding on my side, but because it aggregates the latest state from every reviewer — @nzw921rx, @davidzollo, and your own `2026-07-20` review are still recorded as `CHANGES_REQUESTED` from earlier rounds and haven't been re-submitted as approvals against this head. Worth those reviewers doing a quick re-affirm (or a maintainer dismissing the stale ones) before this clears the actual merge gate, since my approval alone won't flip that flag.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Zeta] Fix stop-with-savepoint hang in DOING_SAVEPOINT [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11489:
URL: https://github.com/apache/seatunnel/pull/11489#issuecomment-5149758741

   Thanks for your contribution for Apache SeaTunnel. 
   
   To facilitate focused communication among contributors, we’ve created a brand-new dedicated #contributors channel in official Slack workspace.
   Slack workspace link: https://s.apache.org/seatunnel-slack
   
   This channel is designed for technical discussions, feature design brainstorming, roadmap sync and direct communication with project core Contributors. Everyone who contributes code, docs, test cases or community ideas is warmly invited to join, share thoughts and collaborate together under The Apache Way. Welcome to join us😄
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Zeta] Fix stop-with-savepoint hang in DOING_SAVEPOINT [seatunnel]

Posted by "agarwalrahul2702 (via GitHub)" <gi...@apache.org>.
agarwalrahul2702 commented on PR #11489:
URL: https://github.com/apache/seatunnel/pull/11489#issuecomment-5235569677

   Pushed `caf39c78d` to fix the merge-blocking CI regression from my previous cleanup commit.
   
   - Removed `import static java.util.concurrent.CompletableFuture.allOf;`.
   - Switched the aggregate wait to `CompletableFuture.allOf(passiveCompletableFutures).join()` using the already-imported `org.apache.seatunnel.engine.common.utils.concurrent.CompletableFuture`.
   - No savepoint behavior changed.
   
   Local verification:
   
   - `./mvnw spotless:check` passes.
   - `git diff --check` passes.
   - `./mvnw -pl seatunnel-ci-tools -Dtest=ImportClassCheckTest#javaUtilCompletableFutureCheck test` passes.
   - `rg -n "import\\s+(static\\s+)?java\\.util\\.concurrent\\.CompletableFuture" seatunnel-engine` returns no matches.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Zeta] Fix stop-with-savepoint hang in DOING_SAVEPOINT [seatunnel]

Posted by "agarwalrahul2702 (via GitHub)" <gi...@apache.org>.
agarwalrahul2702 commented on PR #11489:
URL: https://github.com/apache/seatunnel/pull/11489#issuecomment-5327101322

   Pushed `8f5d15cd3` for the latest review blocker and the latest failed Build run.
   
   What changed:
   - Restored automated coverage for the retry contract requested in the latest review: `SavePointBusySourceTest#testSavepointStartPreconditionFailureCanBeRetriedAfterReady` now verifies that a savepoint rejected before coordinator readiness leaves the job `RUNNING`, and that a later retry after readiness reaches `SAVEPOINT_DONE`.
   - Re-added the retry fixture as `stream_fakesource_retryable_to_console_savepoint.conf`, but made it long-running with many small splits (`500000` rows / `500` splits) instead of using the checkpoint-hostile busy-source fixture. This keeps the job alive for retry without starving savepoint barrier injection.
   - Fixed the Paimon E2E thread-leak failure exposed by the last Build run. In cooperative execution, failed/abandoned tasks were marked done without calling `Task.close()`, unlike the blocking worker path. That can leave sink-owned executors such as `st-multi-table-sink-writer-*` parked after a failing sink test. The cooperative worker now closes failed tasks with the task connector classloader before marking them done, so sink close can shut down its internal executor.
   
   Validation run locally:
   - `./mvnw -pl seatunnel-engine/seatunnel-engine-server spotless:apply` passes.
   - `git diff --check` passes.
   - `./mvnw -pl seatunnel-ci-tools -Dtest=ImportClassCheckTest#javaUtilCompletableFutureCheck test` passes.
   
   I also retried direct `seatunnel-engine-server` compile locally; it is still blocked before useful module verification by this workspace resolving stale/mismatched local reactor artifacts (`JobRestoreInProgressException`, API metadata/signal classes, `RestoreMode`, metric constants, etc.). The full GitHub Build run on `8f5d15cd3` is now the authoritative verification: https://github.com/agarwalrahul2702/seatunnel/actions/runs/32128491990


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Zeta] Fix stop-with-savepoint hang in DOING_SAVEPOINT [seatunnel]

Posted by "agarwalrahul2702 (via GitHub)" <gi...@apache.org>.
agarwalrahul2702 commented on PR #11489:
URL: https://github.com/apache/seatunnel/pull/11489#issuecomment-5023131482

   Thanks for the current-head review. I pushed `35823e758` to address the remaining custom `rows` blocker.
   
   - `FakeDataGenerator` now has a bounded custom-row slice path instead of only the old emit-every-configured-row path.
   - `FakeSourceReader` now applies the same per-`pollNext` upper bound to explicit `rows` as to generated rows.
   - Custom-row progress is persisted in a newly requeued `FakeSourceSplit`: the first custom split starts at row index `0`, continuation splits encode the next custom-row index to emit, and `snapshotState()` therefore records only the not-yet-emitted portion. The reader does not mutate the assigned split object.
   - Added a large custom-`rows` stop-with-savepoint regression config/test so this branch exercises the actual savepoint path.
   - Added a custom-row slice unit test for bounded row generation.
   
   No user-facing config option was added or changed; the new config file is a test resource only, so docs/examples are unchanged.
   
   Verification after this commit:
   
   - `./mvnw spotless:check` passes.
   - `./mvnw spotless:apply` passes.
   - `git diff --check` passes.
   
   Targeted test execution is still blocked by local SeaTunnel build setup before the changed modules are reached: direct module runs fail resolving the local `seatunnel-shade` SNAPSHOT, and `-am` runs fail earlier in `seatunnel-config-shade` because generated/shaded Typesafe Config classes are unavailable in this cleaned workspace.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Zeta] Fix stop-with-savepoint hang in DOING_SAVEPOINT [seatunnel]

Posted by "agarwalrahul2702 (via GitHub)" <gi...@apache.org>.
agarwalrahul2702 commented on PR #11489:
URL: https://github.com/apache/seatunnel/pull/11489#issuecomment-5015627589

   Pushed follow-up commit `e8985e3c` to address the review blockers from the `3a9067c9` head:
   
   1. FakeSource restore correctness:
      - `FakeSourceReader` no longer mutates the assigned split object when batching.
      - Requeued progress uses a new `FakeSourceSplit`.
      - `FakeSourceSplit` identity/hash now uses `tableId` + `splitId`, excluding mutable remaining-row progress.
      - Added a restore regression test that verifies assigned splits are not rediscovered when restored progress has a different remaining row count.
   
   2. Multi-pipeline savepoint finalization:
      - `CheckpointManager#triggerSavePoints()` now consumes `CheckpointCoordinator#startSavepointAndWaitComplete()`.
      - The future observed by `JobMaster` completes after the checkpoint coordinator reaches its terminal state (`SUSPEND` for a successful savepoint), not just after pending-checkpoint ACK.
      - The multi-pipeline regression now asserts terminal pipeline states and verifies that the successful pipeline coordinator reaches `SUSPEND`.
   
   3. Synchronous trigger failure cleanup:
      - `JobMaster#savePoint()` now puts `savepointJob()`, `triggerSavePoints()`, aggregation, and the `savepointFailed()` fallback inside the same guarded operation, so synchronous and asynchronous failures both leave `DOING_SAVEPOINT` through the same cleanup path.
   
   I also updated the PR description so it no longer describes the old “revert to RUNNING / no forced cancellation” behavior.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Zeta] Fix stop-with-savepoint hang in DOING_SAVEPOINT [seatunnel]

Posted by "agarwalrahul2702 (via GitHub)" <gi...@apache.org>.
agarwalrahul2702 commented on PR #11489:
URL: https://github.com/apache/seatunnel/pull/11489#issuecomment-5230199558

   Pushed `d71d4c4e0` to address the two remaining non-blocking review nits from the latest review:
   
   - Added a fine-level log when the aggregate savepoint future completes exceptionally before the method deliberately inspects each pipeline future.
   - Replaced the fully-qualified JDK `CompletableFuture.allOf` reference with a static import, keeping the existing SeaTunnel `CompletableFuture` import unchanged.
   
   No savepoint behavior changed in this commit.
   
   Local verification:
   
   - `./mvnw spotless:apply` passes.
   - `./mvnw spotless:check` passes.
   - `git diff --check` passes.
   
   Targeted engine-server test/compile attempts are still blocked before this module by local repository setup: direct module compile cannot resolve the local `seatunnel-shade` SNAPSHOT from Apache snapshots, and `-am` reaches `seatunnel-config-shade` first and fails on generated/shaded Typesafe Config classes. I also verified the Hazelcast `ILogger#fine(String, Throwable)` API from the local Maven artifact.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Zeta] Fix stop-with-savepoint hang in DOING_SAVEPOINT [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11489:
URL: https://github.com/apache/seatunnel/pull/11489#issuecomment-5316848784

   Thanks @agarwalrahul2702 for the follow-up and the clear summary of your updates. My remaining feedback is in the review — once those points are addressed, please ping me and I'll take another look. This is close!
   
   <!-- streview-comment:177 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Zeta] Fix stop-with-savepoint hang in DOING_SAVEPOINT [seatunnel]

Posted by "agarwalrahul2702 (via GitHub)" <gi...@apache.org>.
agarwalrahul2702 commented on PR #11489:
URL: https://github.com/apache/seatunnel/pull/11489#issuecomment-5254940247

   Pushed `0e3189a30` to correct the failing `SavePointBusySourceTest.testSavepointFailureStopsWholeMultiPipelineJob` path.
   
   Root cause from the failed `438e62754b` unit-test log: the test triggered `savePoint()` immediately after global job status became `RUNNING`, but that can happen before every pipeline checkpoint coordinator has reported all tasks ready. In the failed run, the savepoint request started ~80ms after deployment and hit the `TASK_NOT_ALL_READY_WHEN_SAVEPOINT` path, so both pipelines were force-canceled before the supposed successful pipeline could ever reach `SUSPEND`.
   
   Fix:
   - Added a test helper that waits until every pipeline checkpoint coordinator has `isAllTaskReady=true` before triggering savepoint.
   - Applied that readiness wait to the savepoint tests in this class, replacing the brittle sleep in the failure test.
   - Kept/restored the important partial-success assertion that at least one coordinator reaches `SUSPEND`.
   - Made the successful pipeline fixture longer-running (`row.num = 1000000`) so it cannot naturally drain before the manual savepoint request.
   
   Local validation:
   - `./mvnw -pl seatunnel-engine/seatunnel-engine-server spotless:apply` passes.
   - `git diff --check` passes.
   - `./mvnw -pl seatunnel-ci-tools -Dtest=ImportClassCheckTest#javaUtilCompletableFutureCheck test` passes.
   
   Targeted local engine-server execution is still blocked by the local cleaned workspace generated-shade issue in `seatunnel-config-shade`; the GitHub full-reactor CI run is queued for this head: https://github.com/agarwalrahul2702/seatunnel/actions/runs/31504935952


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org