You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "zhangshenghang (via GitHub)" <gi...@apache.org> on 2026/08/21 14:12:56 UTC
[PR] [Fix][Zeta] Prevent stale task context cleanup [seatunnel]
zhangshenghang opened a new pull request, #11924:
URL: https://github.com/apache/seatunnel/pull/11924
## Description
Fixes #11755.
A task-group completion now cleans up only the execution context created by that same execution generation. A late completion from a previous generation therefore cannot remove the active context, cancellation future, or related resources of a newly restored task group with the same location.
## Testing
- `mvn spotless:check`
- `mvn -pl seatunnel-engine/seatunnel-engine-server -Dtest=TaskExecutionServiceTest#testStaleTaskCompletionKeepsNewExecutionContext test`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Prevent stale task context cleanup [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11924:
URL: https://github.com/apache/seatunnel/pull/11924#issuecomment-5476937193
Thanks for the careful re-check, @JeremyXin — replying to your three points (checked against the current head `4a5e69f1c08a`, which only merges in `dev` to resolve the branch conflicts you flagged; it does not touch `TaskExecutionService.java` at all, so my open Issue 1 from the 2026-08-25 review is still unaddressed there):
1. **Coordination with #11999**: confirmed, this is a real overlap — both fix #11755 by changing `taskDone()` in the same file. They're not identical in scope though: #11999 keeps the original `TaskGroupExecutionTracker` (no `ExecutionGeneration` concept) and adds a `synchronized` block plus a `TaskGroupContext`-identity gate (`finishOwnedResources`/`finishExecutionContext`) around the existing recycle/cancel calls. It doesn't introduce the per-generation `timerFlushFuturesByGeneration`/`taskAsyncFunctionFutureByGeneration` registries this PR adds, so it doesn't inherit this PR's still-open Issue 1 concurrency race — but it also doesn't close the timer-flush/async-function future-leak scenario this PR's own `testTimerFlushCloseOnlyCancelsOwnedExecutionGeneration` regression test targets. This looks like a genuine design fork (simpler synchronized-gate vs. generation-scoped registries), not a trivial duplicate, so it's worth a maintainer call on which shape to keep rather than
trying to reconcile them line-by-line.
2. **Dead code in `closeTimerFlushTaskForGeneration`**: confirmed, you're right. Current head, `TaskExecutionService.java:1202-1208`:
```java
ScheduledFuture<?> future = generationFutures.remove(taskLocation);
if (future == null) {
return;
}
if (future != null && !future.isDone()) { // future != null is unreachable here
future.cancel(false);
}
```
The `future != null` re-check is dead code — the method already returned above if `future` were null. Please simplify to `if (!future.isDone())`.
3. **Are `cancelAsyncFunction`/`cancelTimerFlushForTaskGroup` reached on all completion paths, including exceptions?** Yes — traced `taskDone()` at `TaskExecutionService.java:1646-1690`: both calls sit unconditionally inside the `completionLatch.decrementAndGet() == 0` block, each wrapped in its own `try { } catch (Throwable t) { logger.severe(...) }`, and both run *before* the branch on `ex == null` / `isCancel.get()` that decides FINISHED/CANCELED/FAILED. So they fire on every terminal path — success, cancel, and failure — and a `Throwable` from one doesn't skip the other or block the final state transition. No unbounded-growth risk from a skipped cleanup call on that particular axis.
Branch conflicts are resolved as of the latest commit. My Issue 1 (High, blocking — the same non-atomic `isEmpty()`-then-`outerMap.remove(key,map)` pattern still present in 5 spots, e.g. `TaskExecutionService.java:1184-1185, 1211-1212, 1215-1216`) from the 2026-08-25 review remains open in the current head. Happy to do a fresh full re-review once that lands alongside a decision on how this coordinates with #11999.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Prevent stale task context cleanup [seatunnel]
Posted by "dybyte (via GitHub)" <gi...@apache.org>.
dybyte commented on code in PR #11924:
URL: https://github.com/apache/seatunnel/pull/11924#discussion_r3845819148
##########
seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/TaskExecutionService.java:
##########
@@ -720,17 +748,44 @@ public void cancelTaskGroup(TaskGroupLocation taskGroupLocation) {
* @param task the Runnable to execute
*/
public void asyncExecuteFunction(TaskGroupLocation taskGroupLocation, Runnable task) {
+ asyncExecuteFunction(taskGroupLocation, task, null);
+ }
+
+ public void asyncExecuteFunction(
+ TaskGroupLocation taskGroupLocation,
+ Runnable task,
+ ExecutionGeneration executionGeneration) {
String id = UUID.randomUUID().toString();
logger.fine("accept async execute function from " + taskGroupLocation + " with id " + id);
- if (!taskAsyncFunctionFuture.containsKey(taskGroupLocation)) {
- taskAsyncFunctionFuture.put(taskGroupLocation, new ConcurrentHashMap<>());
- }
+ Map<String, CompletableFuture<?>> groupFutures =
+ taskAsyncFunctionFuture.computeIfAbsent(
+ taskGroupLocation, ignored -> new ConcurrentHashMap<>());
CompletableFuture<?> future =
CompletableFuture.runAsync(task, MDCTracer.tracing(executorService));
- taskAsyncFunctionFuture.get(taskGroupLocation).put(id, future);
+ groupFutures.put(id, future);
+ if (executionGeneration != null) {
+ taskAsyncFunctionFutureByGeneration
+ .computeIfAbsent(executionGeneration, ignored -> new ConcurrentHashMap<>())
+ .put(id, future);
+ }
future.whenComplete(
(r, e) -> {
- taskAsyncFunctionFuture.get(taskGroupLocation).remove(id);
+ Map<String, CompletableFuture<?>> futures =
+ taskAsyncFunctionFuture.get(taskGroupLocation);
+ if (futures != null) {
+ futures.remove(id, future);
+ }
+ if (executionGeneration != null) {
+ Map<String, CompletableFuture<?>> generationFutures =
+ taskAsyncFunctionFutureByGeneration.get(executionGeneration);
+ if (generationFutures != null) {
+ generationFutures.remove(id, future);
+ if (generationFutures.isEmpty()) {
+ taskAsyncFunctionFutureByGeneration.remove(
+ executionGeneration, generationFutures);
+ }
+ }
Review Comment:
Could there be a race here between `generationFutures.isEmpty()` and removing the generation entry? If another async function is added to the same generation in between, it may be added to the same map just before the outer entry is removed, so a later `cancelAsyncFunction()` would no longer be able to find that future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Prevent stale task context cleanup [seatunnel]
Posted by "JeremyXin (via GitHub)" <gi...@apache.org>.
JeremyXin commented on PR #11924:
URL: https://github.com/apache/seatunnel/pull/11924#issuecomment-5475907866
Reviewed the latest head `7c1e180`.
### Merge Conclusion: Request changes
**Blocking: coordination with PR #11999**
Both this PR and #11999 fix the same issue (#11755) by modifying `TaskExecutionService.java`.
We need to figure out how to handle the implementations of these two PRs — please take a look at the details of #11999.
**Additional required fixes:**
1. Dead code in `closeTimerFlushTaskForGeneration` (`TaskExecutionService.java`):
the `future != null` check after the null-return guard is unreachable.
Please simplify to `if (!future.isDone())`.
2. `taskAsyncFunctionFutureByGeneration` and `timerFlushFuturesByGeneration` rely on
`cancelAsyncFunction(location, generation)` / `cancelTimerFlushForTaskGroup(location, generation)`
to be called for cleanup. Please confirm these are reached on all completion paths
(including exception paths), otherwise these maps could grow unbounded under frequent restores.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Prevent stale task context cleanup [seatunnel]
Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang commented on code in PR #11924:
URL: https://github.com/apache/seatunnel/pull/11924#discussion_r3846048150
##########
seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/TaskExecutionService.java:
##########
@@ -720,17 +748,44 @@ public void cancelTaskGroup(TaskGroupLocation taskGroupLocation) {
* @param task the Runnable to execute
*/
public void asyncExecuteFunction(TaskGroupLocation taskGroupLocation, Runnable task) {
+ asyncExecuteFunction(taskGroupLocation, task, null);
+ }
+
+ public void asyncExecuteFunction(
+ TaskGroupLocation taskGroupLocation,
+ Runnable task,
+ ExecutionGeneration executionGeneration) {
String id = UUID.randomUUID().toString();
logger.fine("accept async execute function from " + taskGroupLocation + " with id " + id);
- if (!taskAsyncFunctionFuture.containsKey(taskGroupLocation)) {
- taskAsyncFunctionFuture.put(taskGroupLocation, new ConcurrentHashMap<>());
- }
+ Map<String, CompletableFuture<?>> groupFutures =
+ taskAsyncFunctionFuture.computeIfAbsent(
+ taskGroupLocation, ignored -> new ConcurrentHashMap<>());
CompletableFuture<?> future =
CompletableFuture.runAsync(task, MDCTracer.tracing(executorService));
- taskAsyncFunctionFuture.get(taskGroupLocation).put(id, future);
+ groupFutures.put(id, future);
+ if (executionGeneration != null) {
+ taskAsyncFunctionFutureByGeneration
+ .computeIfAbsent(executionGeneration, ignored -> new ConcurrentHashMap<>())
+ .put(id, future);
+ }
future.whenComplete(
(r, e) -> {
- taskAsyncFunctionFuture.get(taskGroupLocation).remove(id);
+ Map<String, CompletableFuture<?>> futures =
+ taskAsyncFunctionFuture.get(taskGroupLocation);
+ if (futures != null) {
+ futures.remove(id, future);
+ }
+ if (executionGeneration != null) {
+ Map<String, CompletableFuture<?>> generationFutures =
+ taskAsyncFunctionFutureByGeneration.get(executionGeneration);
+ if (generationFutures != null) {
+ generationFutures.remove(id, future);
+ if (generationFutures.isEmpty()) {
+ taskAsyncFunctionFutureByGeneration.remove(
+ executionGeneration, generationFutures);
+ }
+ }
Review Comment:
Good catch. I changed the generation cleanup to use `taskAsyncFunctionFutureByGeneration.computeIfPresent(...)`, so removing the finished future and deciding whether to drop the generation entry now happen atomically for that generation key. This prevents a concurrently registered future for the same generation from being hidden by a stale outer-map removal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Prevent stale task context cleanup [seatunnel]
Posted by "BinTaoMa (via GitHub)" <gi...@apache.org>.
BinTaoMa commented on PR #11924:
URL: https://github.com/apache/seatunnel/pull/11924#issuecomment-5518953625
> 已审阅最新款头部`7c1e180`。
>
> ### 合并结论:请求更改
> **阻塞:与 PR [#11999 的协调](https://github.com/apache/seatunnel/pull/11999)**
>
> [此 PR 和#11999](https://github.com/apache/seatunnel/pull/11999)都通过修改修复了同一个问题 ( #11755`TaskExecutionService.java` ) 。 我们需要弄清楚如何处理这两个 PR 的实现——请查看#11999的详细信息。
>
> **其他需要修复的问题:**
>
> 1. `closeTimerFlushTaskForGeneration`括号内有死代码`TaskExecutionService.java`:空返回保护后的检查无法执行。
> 请
> 简化为。`future != null``if (!future.isDone())`
> 2. `taskAsyncFunctionFutureByGeneration`并`timerFlushFuturesByGeneration`依赖于
> `cancelAsyncFunction(location, generation)`/`cancelTimerFlushForTaskGroup(location, generation)`
> 来进行清理。请确认所有完成路径
> (包括异常路径)都会执行这些操作,否则在频繁恢复的情况下,这些映射可能会无限增长。
>
> 此外,该分支目前存在一些冲突,需要在合并之前解决。
Thanks for the review and coordination.
To clarify, #11999 is my PR and provides the smaller, focused fix for #11755, while #11924 takes a broader generation-scoped cleanup approach.
I’m happy to avoid duplicating work. If #11924 resolves its remaining cleanup issues and merge conflicts, adds sufficient coverage, and is ultimately preferred by the maintainers, I’m willing to close #11999
as superseded.
However, if #11755 needs to be fixed sooner, I suggest merging #11999 first as the focused near-term fix. #11924 could then be rebased and continued separately as follow-up hardening for the broader
generation-scoped resource cleanup.
I’m fine with either direction—the priority is to get a correct and maintainable fix merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Prevent stale task context cleanup [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11924:
URL: https://github.com/apache/seatunnel/pull/11924#issuecomment-5522174791
Thanks for jumping in and clarifying the ownership on #11999, @BinTaoMa — that's helpful context for the maintainers deciding priority here.
From a pure code-review standpoint on **this** PR (#11924), the coordination question doesn't change my open finding. The head is still `4a5e69f1c08a`, unchanged since my last re-review on 2026-08-31/09-01 (the only commit since then just merged `dev` to resolve conflicts and doesn't touch `TaskExecutionService.java`). So Issue 1 from my 2026-08-25 review is still unresolved at the current head — I just re-pulled the file at this exact SHA to confirm:
- `TaskExecutionService.java:1184-1185` — `if (groupFutures != null && groupFutures.isEmpty()) { timerFlushFutures.remove(taskGroupLocation, groupFutures); }`
- `TaskExecutionService.java:1211-1212` — same `groupFutures.isEmpty()` → `timerFlushFutures.remove(...)` pattern inside `closeTimerFlushTaskForGeneration`
- `TaskExecutionService.java:1215-1216` — `if (generationFutures.isEmpty()) { timerFlushFuturesByGeneration.remove(executionGeneration, generationFutures); }`
All three are still non-atomic `isEmpty()`-then-`remove(key, map)` checks: a new timer-flush future can be registered into the same `groupFutures`/`generationFutures` map in the gap between the `isEmpty()` read and the `remove()` call, and that entry then gets silently dropped from the outer map even though it's non-empty at removal time — the same race @dybyte flagged and you already fixed correctly for `cancelAsyncFunction` via `computeIfPresent(...)`. That fix just hasn't been mirrored to these three timer-flush spots yet. This is still High severity/blocking for #11924 regardless of how #11755 priority is resolved between the two PRs.
On the #11999-vs-#11924 sequencing question itself: that's a scope/priority call for the maintainers rather than something I can settle from a diff review — I haven't reviewed #11999's implementation, so I can't vouch for its correctness, only note (per your and @JeremyXin's description) that it takes a narrower `synchronized`-gate approach without the generation-scoped future registries, so it wouldn't inherit this PR's still-open Issue 1. If the maintainers do land #11999 first as the near-term fix, I'd still want #11924 to close out Issue 1 (and ideally land the regression coverage its own `testTimerFlushCloseOnlyCancelsOwnedExecutionGeneration` test is targeting) before merge, whether as a fresh PR or a rebase on top of #11999.
Separately, FYI on CI: the current head's `Build` check on the apache side shows `cancelled` (https://github.com/apache/seatunnel/runs/99465692849, ran ~9h then got cancelled with no steps recorded), and there's no commit status ever posted for `4a5e69f1c08a` (`state: pending`, 0 statuses) — so this head hasn't actually completed a CI run yet, independent of the code review outcome above.
I'll do a fresh full re-review as soon as Issue 1 is addressed and/or the coordination with #11999 is resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Prevent stale task context cleanup [seatunnel]
Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx commented on PR #11924:
URL: https://github.com/apache/seatunnel/pull/11924#issuecomment-5651817773
PR # 11238 has been merged, therefore closed🔥
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Prevent stale task context cleanup [seatunnel]
Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx closed pull request #11924: [Fix][Zeta] Prevent stale task context cleanup
URL: https://github.com/apache/seatunnel/pull/11924
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Prevent stale task context cleanup [seatunnel]
Posted by "dybyte (via GitHub)" <gi...@apache.org>.
dybyte commented on code in PR #11924:
URL: https://github.com/apache/seatunnel/pull/11924#discussion_r3845819148
##########
seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/TaskExecutionService.java:
##########
@@ -720,17 +748,44 @@ public void cancelTaskGroup(TaskGroupLocation taskGroupLocation) {
* @param task the Runnable to execute
*/
public void asyncExecuteFunction(TaskGroupLocation taskGroupLocation, Runnable task) {
+ asyncExecuteFunction(taskGroupLocation, task, null);
+ }
+
+ public void asyncExecuteFunction(
+ TaskGroupLocation taskGroupLocation,
+ Runnable task,
+ ExecutionGeneration executionGeneration) {
String id = UUID.randomUUID().toString();
logger.fine("accept async execute function from " + taskGroupLocation + " with id " + id);
- if (!taskAsyncFunctionFuture.containsKey(taskGroupLocation)) {
- taskAsyncFunctionFuture.put(taskGroupLocation, new ConcurrentHashMap<>());
- }
+ Map<String, CompletableFuture<?>> groupFutures =
+ taskAsyncFunctionFuture.computeIfAbsent(
+ taskGroupLocation, ignored -> new ConcurrentHashMap<>());
CompletableFuture<?> future =
CompletableFuture.runAsync(task, MDCTracer.tracing(executorService));
- taskAsyncFunctionFuture.get(taskGroupLocation).put(id, future);
+ groupFutures.put(id, future);
+ if (executionGeneration != null) {
+ taskAsyncFunctionFutureByGeneration
+ .computeIfAbsent(executionGeneration, ignored -> new ConcurrentHashMap<>())
+ .put(id, future);
+ }
future.whenComplete(
(r, e) -> {
- taskAsyncFunctionFuture.get(taskGroupLocation).remove(id);
+ Map<String, CompletableFuture<?>> futures =
+ taskAsyncFunctionFuture.get(taskGroupLocation);
+ if (futures != null) {
+ futures.remove(id, future);
+ }
+ if (executionGeneration != null) {
+ Map<String, CompletableFuture<?>> generationFutures =
+ taskAsyncFunctionFutureByGeneration.get(executionGeneration);
+ if (generationFutures != null) {
+ generationFutures.remove(id, future);
+ if (generationFutures.isEmpty()) {
+ taskAsyncFunctionFutureByGeneration.remove(
+ executionGeneration, generationFutures);
+ }
+ }
Review Comment:
Could there be a race here between `generationFutures.isEmpty()` and removing the generation entry? If another async function is added to the same generation in between, it may be added to the same map just before the outer entry is removed, so a later `cancelAsyncFunction()` would no longer be able to find that future. Would it be safer to avoid removing the empty generation map here, or make registration/removal atomic per generation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org