You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "abdessalems (via GitHub)" <gi...@apache.org> on 2026/08/09 14:19:50 UTC
[PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
abdessalems opened a new pull request, #11727:
URL: https://github.com/apache/seatunnel/pull/11727
### Purpose of this pull request
Fixes the hang in #11679, where a job cancelled by a downstream privilege check left the
`seatunnel.sh` client blocked indefinitely and burned a full 180-minute CI slot.
I reproduced it in CI and took thread dumps of both the engine and the blocked client while it was
stuck. Dumps and the coordinator log are attached to the issue. The client turned out to be a
symptom — it never receives a terminal result because the engine deadlocks before it can publish
one.
**Root cause.** `TaskGroupLocation` is `{jobId, pipelineId, taskGroupId}` and is reused verbatim
across pipeline restore generations. `TaskGroupExecutionTracker.taskDone()` removes that location's
entry from `executionContexts`, so a late `taskDone()` from an earlier generation can delete the
context that the current generation's `deployLocalTask()` has just installed.
`BlockingWorker.run()` then resolved its class loader through `executionContexts.get(location)`
*before* its `try` block, while `startedLatch.countDown()` sat *inside* it:
```java
ClassLoader classLoader =
executionContexts
.get(taskGroupExecutionTracker.taskGroup.getTaskGroupLocation()) // may be null
.getClassLoaders() // NPE here
.get(tracker.task.getTaskID());
...
try {
startedLatch.countDown(); // never reached
```
A missing context therefore threw before the latch was ever counted down. The exception was
swallowed by the submitting `Future`, so nothing was logged, and `submitBlockingTask()` waited on
`startedLatch.await()` forever — **while holding the `SubPlan` monitor**. The checkpoint-error
thread then blocked at the `synchronized updatePipelineState()` entry and could never move the
pipeline to a terminal state, so `JobMaster`'s completion future never completed and the client
waited on it indefinitely.
From the dumps:
```
"seatunnel-coordinator-service-8" WAITING (parking)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at ...TaskExecutionService.submitBlockingTask(TaskExecutionService.java:405)
...
at ...SubPlan.updatePipelineState(SubPlan.java:397)
- locked <0x00000000f6d2c470> (a SubPlan)
"hz.main.generic-operation.thread-96" BLOCKED (on object monitor)
at ...SubPlan.updatePipelineState(SubPlan.java:345)
- waiting to lock <0x00000000f6d2c470> (a SubPlan)
at ...SubPlan.handleCheckpointError(SubPlan.java:644)
```
### Does this PR introduce a user-facing change?
No behaviour change in the normal path. A deployment whose execution context has disappeared now
fails with a clear `IllegalStateException` naming the task group, instead of hanging silently.
### How was this patch tested?
Added `TaskDeployStaleContextRaceTest`, which asserts the narrow contract that prevents the hang:
deploying a task group must return even if its execution context disappears while the deployment is
in flight. It does not assert *how* it returns — success or failure both satisfy it. Only a
deployment that never returns is the defect.
Run on the JDK 8 leg, same commit, test alone versus test plus fix:
| | Result |
|---|---|
| Test on unpatched engine | `Tests run: 1, Failures: 1` — *"deployTask did not return within 30s on iteration 0"*, 32.8s |
| Test with this fix | `Tests run: 1, Failures: 0` — 30 iterations in **3.9s** |
The original `PaimonWithS3IT` hang reproduced roughly 1 in 6 runs before this change.
### A note on the fix
`startedLatch.countDown()` deliberately stays **before** `init()`. Moving it into `finally`
unguarded would make `submitBlockingTask()` wait for tasks to *finish* rather than *start*, which
would deadlock every long-running task. The signal is guarded by a `startSignalled` flag so each
worker releases the latch exactly once, including on the failure path.
### Scope, and a follow-up
This fixes the **hang**, not the race that triggers it. A stale `taskDone()` can still remove a
live execution context; the deployment now fails loudly instead of wedging the engine. Making that
removal conditional — so a tracker from an earlier generation cannot delete a newer entry — is a
separate change with its own risk profile, and I would rather propose it on its own than bundle it
here. Happy to open it as a follow-up if maintainers agree with the direction.
Marked as draft for that reason: the containment is verified, the locking design decision is yours.
Related: #11718 keeps the CI blast radius small and is complementary — it does not overlap with this
change.
### Check list
* [x] Code changed are covered with tests, or it does not need tests for reason
* [x] If any new Jar binary package adding in your PR, please add License Notice according
[New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md)
* [x] If necessary, please update the documentation to describe the new feature.
* [x] If you are contributing the connector code, please check that the following files are updated:
* [x] Update the `docs/en/connector-v2` and `docs/zh/connector-v2` for the connector.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5678488297
Thanks @abdessalems — good catch, and you're right.
I re-pulled the diff to double check (`gh pr diff 11727 --name-only`): this PR only touches `TaskExecutionService.java`, `TaskDeployStaleContextRaceTest.java`, and `TaskExecutionServiceTest.java`, and the only test method it *adds* is `testMissingTaskClassLoaderFailsFast`. `testStaleTaskDoneCleansOnlyOwnedGenerationResources` and the `finishOwnedResources()` Javadoc I was describing to @SEZ9 are both present on the current head only because they landed via #12238 and are already on `dev` — neither is part of this PR's own diff. My answer was accurate about what's true of the current head as a whole, but I should have been explicit that those two specific points describe #12238's contribution, not this PR's. Appreciate you tightening up the attribution so the record is clean for whoever does the final merge pass.
This doesn't change my last full review's conclusion — what #11727 actually contributes is still just the `BlockingWorker.run()` latch-release fix in `TaskExecutionService.java` plus its own two tests, and I still consider it ready to merge pending @SEZ9's pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5706506926
Thanks for the ping, @abdessalems — good question, and easy to check precisely since I can pull the actual job logs rather than guess from the red X.
I pulled both `engine-v2-it` job logs from the rerun on `2dc21468105`:
- **JDK 8** (`Run / engine-v2-it (8, ubuntu-latest)`): `SplitClusterFaultToleranceIT.testStreamJobCancelResolvesWhenWorkerCrashesBeforeCancelAck` failed with `expected: <CANCELED> but was: <FAILED>` — this is exactly the same failure I traced on `ce142f830` and reported back then. It's a pre-existing race on `dev` itself: when a worker is lost mid-cancel, the vertex sometimes resolves to `FAILED` instead of `CANCELED`. Tracking fix: apache/seatunnel#12311 ("Resolve a CANCELING vertex to CANCELED, not FAILED, when its worker is lost") — **still open, not yet merged into `dev`**.
- **JDK 11**: `SplitClusterFaultToleranceIT` actually passed this time (0 failures, 1 skipped). Instead, `BackpressureSlowSinkIT.testCheckpointsKeepCompletingUnderSustainedBackpressure` failed: `expected at least 3 additional checkpoints to complete during the 90s sustained backpressure window, only observed 0`. This is a different, also pre-existing `dev` flake — a source checkpoint-lock starvation issue where the reader thread can beat `triggerBarrier` on an unfair monitor. Tracking fixes: apache/seatunnel#12316 (engine-side fix) and #12313 (test determinism follow-up) — **both still open, not yet merged into `dev`**.
So to directly answer your question: it's not a new failure and not something introduced by this PR's diff (which only touches `BlockingWorker.run()`'s class loader resolution and has no relationship to checkpoint-barrier injection or cancel/failover state resolution). Both are known, already-diagnosed `dev`-level flakes with fixes already up for review.
One important nuance though: since #12311/#12316/#12313 haven't landed on `dev` yet, **syncing this branch with the latest `dev` right now would not make these two pass** — you'd likely just trade one flaky leg for the other on a rerun, as you've already been seeing across the last few reruns. I'd treat both as known/tracked flakes that don't block this PR, and I don't think there's anything actionable for you to change here. If you want, a plain rerun of just those two failed legs is reasonable; otherwise this is fine to leave as-is from a source-review standpoint — my source-level review on this head stands.
The other reds you mentioned (`all-connectors-it-2/6/7`, `paimon-connector-it`) I'd bucket the same way unless a rerun shows a stack trace that actually touches something in your diff — happy to take a look at those logs too if they keep failing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5732574832
@DanielLeens — pushed the F7 breadcrumb in 06c2ef03e. The race test's class
Javadoc now states explicitly that the redeploy-vs-taskDone half lives in
deployLocalTask() and the tracker teardown, that neither is touched here, and
that it's #12238's design and tracked against it — so nothing in this test reads
as covering it.
Thanks for re-checking the diff independently rather than taking my grep on
faith.
@SEZ9 — that closes the F7 coverage thread from my side. F2 is the one item left
for you to confirm against the diff.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5466031869
Thanks for confirming, @abdessalems — that matches what I'd have guessed: a lossy merge-conflict resolution rather than a deliberate call, and 13s locally for 30 iterations lines up with what I'd already seen in the fork's CI logs for the pre-cut version, so there's no timeout pressure pushing the other way either.
Restoring `ITERATIONS` to 30 fully closes Issue 4, which was the only open item from my last pass — Issues 1-3 are the pre-existing carried-over Medium/Low follow-ups (not new blockers, and not specific to this head). Once that one-line test change lands I'll take a quick fresh look, but I don't expect it to change anything: it's test-only, doesn't touch `TaskExecutionService.java`, and my "Ready to merge after fixes" conclusion already accounted for this being the last thing to land.
Good to hear the `@waterWang` sync is already on your radar too — that was the other loose end I wanted acknowledged before merge, so we're in good shape there. Nice work carrying this all the way through.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5437621792
Pushed a fix for the cancelled unit-test legs, @DanielLeens.
They weren't cancelled by the connector-IT failures — they hit the 90-minute job timeout. Six existing tests in TaskExecutionServiceTest were ending up FAILED instead of FINISHED and then waiting out their full Awaitility timeout.
The cause was the getClassLoaders() guard from the last round. The guard itself is correct, but the test helper was only registering a class loader for the first task in a group. So tests deploying two tasks left the second one without an entry. That was previously hidden because the missing entry just resulted in a null context class loader. deployTask() registers one per task, so I updated the helper to mirror that.
Locally, TaskExecutionServiceTest and TaskDeployStaleContextRaceTest are now 17/17 green, and the tests complete in about 16s instead of 80s.
I also checked the other two candidates: removing the synchronized (TaskExecutionService.this) block didn't change the failures, while removing the recycleClassLoader call on the stale path caused a hang instead.
@waterWang — also worth noting for #11757: its unit-test job was skipped, so this issue wouldn't have been caught there either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5248822096
@nzw921rx Thanks for taking a close look — glad to have another set of eyes on this.
**On the ordering concern:** I traced this specifically and I don't think there's a race here, for a simple reason: `startSignalled` is a plain local variable declared inside `BlockingWorker.run()` (`boolean startSignalled = false;`), not a field on the worker or on any shared object. `submitBlockingTask()` creates one *fresh* `BlockingWorker` instance per task and submits each exactly once to the executor, so `run()` executes precisely once per instance, on a single thread, and nothing outside that one method activation ever reads or writes `startSignalled`. The two touches you flagged — `startedLatch.countDown(); startSignalled = true;` on the normal path, and `if (!startSignalled) { startedLatch.countDown(); }` in `finally` — are just two sequential statements in the same stack frame on the same thread, with a plain assignment in between them that can't throw. So there's no interleaving window for another thread to observe `startSignalled == false` after the countdown has a
ctually happened, and no JMM/visibility concern either, since the variable is thread-confined by construction. I don't see a double-release or missed-release path here.
**On reusing `result`/a lifecycle state instead of the boolean:** I like the instinct, but I don't think `result` can carry this signal without changing behavior. `result` (the `ProgressState` from `t.call()`) stays `null` until *after* `t.init()` and the first `call()` complete, whereas the latch is deliberately released *before* `init()` runs (see the comment right above the countdown) — that's the whole point of this fix: the deployer must be released once the worker has resolved its context and is about to start, not once the task has produced its first result. `ProgressState` also has no "started" concept today (only `isDone()`/`isMadeProgress()`), so `result.isStarted()` would require adding new state to that enum. Tying the release to `result` would push it past `init()`, which risks reintroducing a hang if `init()` itself blocks — exactly the class of bug #11679 is about. So I think a dedicated flag is the right shape here; it isn't really standing in for a reusable ta
sk-lifecycle state, it's tracking something narrower ("has this specific worker already released its slot of the latch"), which existing state objects don't model.
**On the name:** agreed that `startSignalled` reads as "has the task started" when it really means "has this worker already counted down its latch slot." `startLatchReleased` (or similar) would be clearer. That's a good, low-risk follow-up — happy to see it done as a quick rename here or in a fast-follow, purely cosmetic, no behavior change either way.
Net: I don't consider this a blocker for merge, but the naming clarification is a nice-to-have I'd support taking in this PR if it's an easy touch-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5578537821
@SEZ9 thanks for the careful read, and for agreeing on Blocker 1's tracking split.
**On the "cut off" concern:** I checked the stored review body via the API rather than the web renderer, and it is not actually truncated — the sentence after "either submit call" reads in full: "(e.g. `RejectedExecutionException` from an executor mid-shutdown, exactly the kind of race a failover/restore window produces) leaves the `executionContexts`/`cancellationFutures` entries in place with no rollback (this is SEZ9's F3)." followed by the rest of the Blocker 1 writeup through to the "Once Blockers 1 and 2 are addressed..." closing paragraph. So the record is complete as posted; if it displayed cut off, that was a rendering artifact rather than a content gap. Pasting the exact continuation here for the record per your ask, in any case.
**On whether this PR's ownership changes depend on the leaked entry being absent:** I traced `finishOwnedResources()`/`finishExecutionContext()` fresh against this head (`18e821c3d`, unchanged from `370f175e879`) to answer this precisely rather than assume it either way.
`finishExecutionContext()` (`TaskExecutionService.java:1594-1600`) does `executionContexts.remove(taskGroupLocation, ownedContext)` — a compare-and-remove keyed on the exact context object this tracker owns. That call is safe against *any* current map value: if the map holds something other than `ownedContext` (a leaked entry, a newer generation's context, or nothing at all), the remove simply returns `false` and `finishOwnedResources` takes the "stale" branch (`:1571-1576`), preserving whatever is there. So no, the ownership logic's own correctness does not assume the leaked entry is absent — it was written to tolerate an unexpected map value.
What the leak *does* break is upstream of this: if `deployLocalTask` leaks an `executionContexts` entry for a task group whose submit calls threw before any task actually started, no `TaskGroupExecutionTracker` for that generation ever reaches `taskDone()`/`finishOwnedResources()` (the tasks never ran to call it), so that leaked entry is never moved to `finishedExecutionContexts` by this PR's own logic either. And because `deployTask`'s outer guard (`:506`) treats any live `executionContexts` entry as "already active" regardless of whether it's a real running generation or a leaked one, no subsequent redeploy for that `TaskGroupLocation` can even create a new generation to be tracked. So: this PR's `ownedContext`/`finishOwnedResources` path is not incorrect in the presence of the leak, but it is starved — it never gets a chance to run again for that location once the leak occurs, because #12164's bug blocks the very redeploy that would produce the next generation for it to own.
Worth noting explicitly, as you said, so the follow-up on #12164 doesn't accidentally get scoped as "the ownership fix's problem" or vice versa — they're independent bugs that compound in exactly this one failure window.
**Blocker 2 (stale-generation's own async/timer futures not cancelled):** agreed, still open and still a blocker for this PR specifically — it's this PR's own diff, not baseline-shared with #11757 the way Blocker 1 is.
**On the rest (F1/F3, F2, F5, F6, F7, F8):** confirmed still open — the head is genuinely unchanged (`370f175e879` through `18e821c3d` are all empty CI-retrigger commits, no source diff), so nothing has moved since your 08-28/08-31 findings.
@davidzollo — to summarize the concrete ask so the next push addresses everything in one pass: please cover Blocker 2/F4 (cancel the old generation's own async/timer-flush futures in the stale-taskDone branch), F1/F3 (either take the deployment monitor for the redeploy-vs-taskDone race on `cancellationFutures`, or fix the javadoc and the race separately — they currently disagree), F2 (`BlockingWorker` should resolve its context from the tracker's `ownedContext`, not the shared `executionContexts` map), and F5 (guard the `put` so it can't silently overwrite a still-live older generation's entry/future). F6-F8 can be legitimate follow-ups if you'd rather keep this push focused — just say so explicitly in the PR description so it's not read as an oversight. Once that lands, I'll do the full from-scratch re-review as usual.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5633066504
@SEZ9 Following your status check on `f1e02f942` — no new source-side blocker from my side beyond what I already flagged above: the PR description now covers #12164/#12224 (closes Issue 2 from my last review), and the fork's `Build` run for this head finished `FAILURE` in `connector-rocketmq-e2e`, which looks unrelated to this PR's `TaskExecutionService` diff but should be rerun before anyone calls CI green.
F1/F3/F7/F8 remain your thread with @abdessalems as I noted last time — I don't have anything to add there beyond the reflection-based field access I already verified at `TaskExecutionServiceTest.java:782-789`. I'll hold for your and @abdessalems's resolution on the F8 `get`/`remove` question, the F7 deferred-test note in the description, and a clean CI run on this head.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5249420202
@nzw921rx Thanks for the review - good catch on the name. Renamed it to
`startLatchReleased` in 79eebc5. You're right that `startSignalled` reads as
"has the task started", when what it actually tracks is whether this particular
worker has already counted down its slot of the latch.
On modelling it through `result` instead: I looked at that and I don't think it
works here. `result` stays null until `t.call()` returns, but the latch is
released before `t.init()` on purpose - the deployer should be freed once the
worker has resolved its context and is about to start, not once the task has
produced a first result. Moving the release after `init()` would put a blocking
`init()` back in the path that #11679 is about. `ProgressState` also has no
"started" notion today, only `isDone()`/`isMadeProgress()`.
The rename is the only change - no behaviour difference. Re-ran the module on
Ubuntu/JDK 8 after it: `TaskDeployStaleContextRaceTest` 2/2, BUILD SUCCESS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5237307964
Thanks for the thoughtful follow-up, and glad the taskDone() point landed clearly - that was the part I most wanted to make sure didn't get lost, since it's a second, independent bug that happened to be hiding behind the same NPE as the startedLatch issue.
On your question for Issue 1: I'd lean toward including the null check in recycleClassLoader() in this same PR rather than splitting it out. The reason is that this PR is precisely what makes that path more reachable - before your fix, the pre-try NPE meant taskDone() (and therefore recycleClassLoader()) was frequently never reached in the stale-context race scenario, so the exposure was accidental and inconsistent. After your fix, taskDone() is guaranteed to run from the finally block, so the same race that motivated this PR becomes more likely to reach the un-null-checked get() in recycleClassLoader(). Since the fix and the newly-increased exposure are causally linked, keeping them in one PR tells a cleaner story for anyone reading the history later, and it's a small, low-risk, purely defensive addition (null check, log a warning, return) so it shouldn't meaningfully grow the diff's risk surface. That said, this is genuinely non-blocking - if you'd rather keep this PR minimal an
d ship the null check as an immediate fast-follow, that's a fine call too. I'll leave the choice to you.
Sounds like a solid plan overall: test comment clarification, Javadoc update on the BlockingWorker.run() list, your call on Issue 1, and a filed follow-up for the underlying stale-taskDone()/executionContexts race linked back here and to #11679. Once those land and you take the PR out of draft, I'll do a full pass over the updated diff. Nothing here changes my earlier conclusion - this is a solid, well-verified fix for a real hang, and I'm looking forward to seeing it merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5252911866
Thanks everyone for the reviews and for taking the time to verify the fix. The latest head has two approvals, and all four unit-test legs are green, including the engine-server regression tests. The remaining connector failures are unrelated to this change, as confirmed in the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5578784666
Pushed a fix at `e66911c0c` (maintainer-edit, as davidzollo). Going through your asks one by one:
**Truncation claim**: checked the stored review body via the API end-to-end - it is not actually truncated. The sentence after "either submit call" reads in full: "(e.g. `RejectedExecutionException` from an executor mid-shutdown, exactly the kind of race a failover/restore window produces) leaves the `executionContexts`/`cancellationFutures` entries in place with no rollback (this is SEZ9's F3)." Whatever you saw was likely a rendering artifact on GitHub's side, not a data loss on ours - happy to paste any other section if something still looks cut off to you.
**Does the ownership fix depend on Blocker 1's leak being absent?** No. `finishExecutionContext`'s `executionContexts.remove(taskGroupLocation, ownedContext)` is a pure reference-identity compare-and-remove; it doesn't care whether some *other*, unrelated deployment attempt elsewhere leaked an entry. And if Blocker 1's leak *did* occur for this exact location, `deployTask`'s `containsKey` guard would already be permanently blocking any new deployment there (that's the whole point of Blocker 1's severity) - so no new generation would ever reach `asyncExecuteFunction`/`registerTimerFlushTask` to register anything that could conflict with this fix's tagging. The two are orthogonal.
**Blocker 2/F4 (fixed)**: `taskAsyncFunctionFuture`/`timerFlushFutures` entries are now tagged with the `TaskGroupContext` active at registration time (`OwnedFuture`, `TaskExecutionService.java`). `finishOwnedResources`'s stale branch now calls `cancelOwnedAsyncFunctionsInPlace`/`cancelOwnedTimerFlushTasksInPlace`, which cancel and remove only entries tagged with `ownedContext`, leaving anything tagged with a different (necessarily newer) context strictly untouched. `registerTimerFlushTask`'s existing exact-`TaskLocation` replace-and-cancel logic didn't need the tag (re-registering for the same `TaskLocation` is always a legitimate replacement, regardless of generation) - only wrapped for the type change.
**F1/F3 (redeploy-vs-taskDone race on `cancellationFutures`)**: I checked reachability before deciding how to fix this - grepped the whole repo for callers of the unguarded public `deployLocalTask()` overload; the only caller anywhere is `TaskExecutionServiceTest`. Production RPC always goes through `deployTask()`, whose `synchronized(this)` block wraps the entire `containsKey` check + `deployLocalTask()` call, so the race can't fire in production today. Given that, I took your documentation option: both `deployLocalTask()`'s and `finishOwnedResources()`'s javadocs now state the actual contract precisely (the guard lives in `deployTask`'s caller-side block, not internally), rather than introducing new runtime guard logic whose semantics I'd have to invent for a path nothing production calls.
**F5 (plain put overwrites a live entry)**: same root cause and same fix as F1/F3 - it's the `deployLocalTask()` overload's lack of self-guarding, now documented as such.
**F2**: already fixed at this head - `BlockingWorker.run()` resolves via `taskGroupExecutionTracker.ownedContext` (not the shared map), with a comment explaining why. I think this is stale from an earlier round; let me know if you're seeing a different call path still using the shared map.
**F6/F7/F8**: F6 remains structurally neutralized by `deployTask`'s `containsKey` guard, per my last assessment. F7 (test coverage) - added `testStaleTaskDoneCancelsItsOwnAsyncAndTimerFutures`, specifically covering the redeploy-vs-taskDone commingled-bucket scenario (old generation's own async/timer entries alongside a replacement generation's, verifying only the stale ones get cancelled). F8 (dead-code branch, `finishExecutionContext` doc gap) still stands as a documentation-only nit - happy to fold in as a follow-up if you'd like it in this PR too, just say so.
Verified locally (compile + full `TaskExecutionServiceTest` suite, 18/18 passing, including both stale-generation tests) before pushing, given how central this class is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5553637636
CI note on the retrigger run (`5a442e45b`, still in progress, 28/93 jobs done): two jobs already red, both traced and both unrelated to this PR's `TaskExecutionService` diff — (1) `transform-v2-it-part-1 (8)`: `TestFilterRowKindIT.testFilterRowKindMultiTable` on the Flink 1.18.0 leg, `row num :25 MIN_ROW` / `:125`,`:150 MAX_ROW` — the static-counter defect tracked in #12116 (fourth PR to hit it); (2) `all-connectors-it-4 (8)`: `AzureCosmosDBSourceIT` — both tests failed reading back from the Cosmos DB emulator with `CosmosException: Service is currently unavailable … Network timeout or connectivity failure`, i.e. the emulator container went unresponsive; first sighting, on the watchlist. Will retrigger once the run completes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5594972608
@DanielLeens thanks for checking the stored body via the API — agreed, if the full "either submit call …" sentence and the closing "Once Blockers 1 and 2 are addressed…" paragraph are there, the record is complete and what I saw was a rendering artifact. No further ask on that.
On the ownership-dependency question, I have the mirror problem on this side: your comment as it reaches me ends mid-sentence at "`finishExecutionContext()` (`TaskExecutionService.java:1594-1600`) does `executionContexts.remo`", so I don't have the conclusion from your fresh trace against `18e821c3d`. Rather than assert the answer for you, could you re-post the continuation (plain text is fine)? Specifically: does the `remove(key, value)` path in `finishExecutionContext()`/`finishOwnedResources()` tolerate a leaked entry being present, or does it only behave correctly when the entry is absent? That determines whether the leaked-entry point stays folded into the tracking split we agreed on for Blocker 1, or needs a rollback in this PR alongside the F5 overwrite concern.
<!-- streview-comment:912 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5584164605
Thanks for the re-review @DanielLeens, and @davidzollo for pushing the
async/timer cancellation fix.
On CooperativeTaskWorker — I'd rather defer that one than widen this PR again.
It's the same generation-scoping pattern as BlockingWorker but on the
thread-share path, and it only bites under a non-default configuration. Happy
to file it as a follow-up issue and pick it up next, alongside #12164, unless
you'd prefer it closed here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5741385806
Thanks @davidzollo for the approval and for merging dev in.
CI on a80834ebf: all four unit-test legs green. engine-v2-it is still red on
both legs, but that's the SplitClusterFaultToleranceIT race @DanielLeens traced
to dev's own code — tracked in #12311, still unmerged, so the dev sync wouldn't
clear it. The rest are the usual connector ITs.
One procedural thing: the PR still shows CHANGES_REQUESTED from @DanielLeens
(09-07) and @SEZ9 (08-28). Both were resolved through later COMMENTED updates
rather than a formal state change, and @DanielLeens has said he can't update his
own since his account is comment-only. Could you dismiss those when you're
happy, so the review gate clears?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5806299139
@abdessalems — thanks for the follow-up, and sorry for the delay.
On the dismissal ask: I'd rather not change any review state based on the discussion alone. Could you share what the PR currently reports for its aggregate review decision and merge state, so I can confirm what is actually gating the merge?
On the review scope: I haven't yet verified myself that this PR's content is unchanged between 06c2ef03e and a80834ebf; I'll re-check that diff. A few items still look open to me:
- F2 and F7 are still pending verification per your note, so I'm not treating them as closed yet.
- F4 (stale-cleanup branch not cancelling the old generation's async functions / timer-flush tasks) hasn't been addressed in the thread — could you say where that stands?
- For F1/F3/F5/F6/F8, I'd like to confirm against the current diff whether those code paths are introduced by this PR or only present on `dev` before treating them as out of scope.
On CI: could you link the failing jobs on a80834ebf so we can separate the known flake from any connector IT failures?
Once the diff and the findings above are settled and CI is green, we can move forward.
<!-- streview-comment:1266 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5411498619
@DanielLeens @SEZ9 — first, sorry for all the back and forth on this one, and for taking up so much of your time over the last few days. The reviews have been genuinely detailed and I've learned a lot from them, so thank you for the patience.
Quick status and one question before I push.
The class loader guard in BlockingWorker.run() is done — a missing per-task entry now fails fast with an IllegalStateException instead of silently installing a null context class loader.
On the eviction item, I started with the change as described, having taskDone() capture the removal and hand that instance to recycleClassLoader() so it stops re-resolving by key. That does fix a real inconsistency, since previously the recycle and the archive could act on two different contexts. But working through it I don't think it fully closes the race on its own: if an older generation's taskDone() runs after a newer one has already installed its context under the same reused location, the remove() hands back the newer context, and we'd recycle its class loaders while it's still running.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5462628507
Good catch on the iteration count, @DanielLeens — that wasn't intentional.
410b47409 is a merge of dev into the branch and ITERATIONS got dropped from
30 to 5 in the conflict resolution rather than as a deliberate change. I'll
restore it to 30; the two tests run in about 13s at that count locally, so
there's no runtime reason to lower it.
Thanks for the thorough passes on this one throughout — and agreed on syncing
with @waterWang so the identity-checked removal lands once rather than twice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5659338078
@SEZ9 — both are already up: the test Javadoc update is in 01a705240, and the PR
description was updated the same day. Ready for your pass whenever you have time.
One correction to my earlier note on F7, so you don't find it as a surprise: none
of the ownership-internals reflection remains, but TaskDeployStaleContextRaceTest
does still reflect into executionContexts (line 276) — that's how the remover
thread simulates an entry disappearing mid-deploy. The TaskExecutionServiceTest
additions don't use reflection. Given your earlier call on F7 to keep reflection
rather than widen the production API, I've left it as is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5646715055
@davidzollo @DanielLeens @SEZ9 — now that #11757 is closed in favour of #12238,
worth settling how that interacts with this PR before either merges.
#11727 carries three commits cherry-picked from #11757 (588314d9c, 1fed68145,
912e8bd4c) plus the later work on top. #12238 solves the same problem a different
way — re-keying the futures by TaskGroupContext rather than the OwnedFuture
tagging here. If both land we'd have two ownership models in the same methods.
This PR's original scope was just the unconditional startedLatch release for
#11679; the ownership model was cherry-picked here in August at @DanielLeens's
and @davidzollo's suggestion, before #12238 existed.
So: would you prefer #12238 lands first and I strip this back to the latch fix
for #11679? Or this merges first and #12238 rebases? Either works for me — I'd
rather ask than have the two conflict.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5691955264
@SEZ9 — all three, checked against the diff rather than from memory:
1. F2 — confirmed. TaskExecutionService.java:1294:
ClassLoader taskClassLoader =
taskGroupExecutionTracker.context.getClassLoaders().get(t.getTaskID());
That field is @NonNull and pinned in the tracker's constructor, so a reused
TaskGroupLocation cannot hand this worker another generation's loader. The
shared-map lookup that used to sit before the try block is gone, and with it
the null-context branch.
2. F7 (reflection) — I couldn't find a non-reflective seam that still races the
deployment: the map is private, and the public surface (deployTask,
cancelTaskGroup) only removes an entry as part of tearing the task group down,
which would stop the deploy rather than race it. A package-private accessor
would widen the production API for test-only visibility, which is what you
ruled against earlier. So I've kept the reflection and added a Javadoc on
executionContextsOf() explaining exactly that — pushed in 2dc214681.
3. F4 / F6 / F8 — not touched by this diff. grep over the diff gives zero
occurrences of finishExecutionContext, finishOwnedResources,
synchronized (TaskExecutionService.this) and cancelOwnedAsyncFunctionsInPlace.
The whole diff is three hunks: one import removal, and two inside
BlockingWorker.run(). So those belong with F1/F3/F5 against dev.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5365075810
Thanks a lot for the thorough re-review, @DanielLeens, and for confirming that there are no blockers.
Agreed on keeping #11727 as the main PR. Since #11910 is the duplicate, feel free to close it as superseded.
@davidzollo @zhangshenghang — the PR is ready from my side. When you have a chance, could you please take a look and merge it if everything is in order?
Thanks again everyone for the reviews and help throughout this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5358600737
@DanielLeens Thanks for the detailed review, and for catching the second null-safety gap in
taskDone() — that one was a real hole, not a nitpick.
I've merged the latest dev into this branch and pushed, so #11727 is now on a
current base and CI is running again. The push went through from my side since
it's my fork — "Allow edits by maintainers" is already enabled here, so the 403
wasn't a settings problem on this PR.
If CI comes back green, #11910 can be closed and we can keep everything on this
one. Happy to do it the other way round if you'd prefer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5552701678
## CI triage on `a30a0bd7b` (run still in progress — no retrigger until it completes)
Three jobs have failed so far. Each traced from its own log rather than assumed:
1. **`doris-connector-it (8)` — `DorisErrorIT.testDoris` @ `DorisErrorIT.java:85`.** The pinned-frame assertion; stderr shows `DorisStreamLoad.stopLoad` → `Doris-01 stream load error` ← `UnknownHostException: doris_e2e`. Docker teardown race in dev's test (same as #11503). Source fix: #12114 (test-only).
2. **`doris-connector-it (11)` — `DorisSchemaChangeIT.testDorisWithSchemaEvolutionCase` @ `:255`.** `assertIterableEquals` on `select * … order by id` differed at row `[9]`: source id `164`, sink id `173` — the sink was still missing the 164–172 range when the 60 s post-DDL window expired. The same log shows 60× `Doris-01 stream load error` retries during the schema-evolution phase, i.e. stream-load retry lag, not data loss (the test cancels the job on timeout so final convergence is unobservable). Timing flake in dev's test; first sighting — on the recurrence watchlist.
3. **`all-connectors-it-7 (11)` — `PostgresCDCIT.testPostgresCdcSnapshotOnlyAndCommittedOffsetStartupModes` @ `:625`.** After `restoreJob` of the committed-offset job (`7942339021350665806`) the job deployed and ran normally — `SCHEDULED → DEPLOYING → RUNNING`, both tasks started, slot active, WAL streaming — for the full 3-minute window, but the post-restore insert (id 15) never reached the sink, and the test then cancelled it (`end with state CANCELED`). That is exactly open issue #11847 (committed-offset restore skips a row when the slot LSN advances past it), with fix #11864 still open. Relevant to this PR's review: the `BlockingWorker` start-latch change is on the *failure-to-start* path, which never fired here — tasks deployed and started cleanly — so this is not attributable to the diff.
None of the three touch files in this PR's diff (`TaskExecutionService` + tests). Plan: once the run completes I'll retrigger; 1 clears when #12114 lands and is synced, 3 will keep recurring intermittently until #11864 lands.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5627599899
@abdessalems Thanks for the update. Two quick notes rather than a full re-review, since I already reviewed this exact head (`f1e02f942`) in my last comment and nothing in the source changed since then:
- **PR description**: confirmed — it now includes the "Scope, and follow-ups" section linking #12164 and #12224, and the stale "Marked as draft" line is gone. That closes Issue 2 from my last review. Thanks.
- **CI**: the fork's `Build` run for `f1e02f942` has since finished with `FAILURE` (it was still queued when I posted my last review). The failing job is `connector-rocketmq-e2e` (`E2E : Connector V2 : Rocketmq`), not anything in `seatunnel-engine-server` — that module and its own tests all completed `SUCCESS` in the same run before the Rocketmq E2E job failed later in the reactor. So this doesn't look related to this PR's diff, but flagging it since "CI green" isn't accurate to claim right now — worth a rerun of just that job to confirm it's the usual flake rather than something new.
On F7 and the local `20 tests, 0 failures` note: I'll leave those to @SEZ9 since they're the open thread there and I don't have anything to add beyond what I already verified in my last review (the reflection-based field access in the new tests, confirmed via `TaskExecutionServiceTest.java:782-789`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5232033945
DONE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5233707712
@davidzollo Thanks David, really appreciate the feedback and the invitation. I'll definitely keep contributing and look forward to collaborating with everyone
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5413195809
> @DanielLeens @SEZ9 The part that actually prevents that is the remove(key, value) identity check against the context pinned at deploy time, which needs the tracker to hold the context it owns. That looks like the same ownership model as ownedContext in #11757, so rather than write it twice or risk conflicting logic with @waterWang's work, I'd rather confirm the intended split with you. Would you prefer the ownership fix here in #11727, with #11757 rebasing on top? Or should #11727 land with the narrower change and leave the identity check to #11757? Happy to go either way, I just didn't want to guess. I'll hold the eviction change and the cross-generation test until you've had a chance to weigh in.
Thanks for calling this out. I agree with your analysis: moving remove() before recycleClassLoader() is not sufficient by itself, because a stale generation can still remove the newer generation's context if that context has already been installed under the reused TaskGroupLocation.
My preference is to fix the ownership model in this PR (#11727): let the tracker hold the TaskGroupContext it owns, use executionContexts.remove(taskGroupLocation, ownedContext), and only recycle/archive the context when that identity-checked remove succeeds. Then #11757 can rebase on top of this PR and focus on any remaining broader cleanup/follow-up.
I would not merge #11727 with the narrower change and leave the identity check entirely to #11757, because this PR itself makes taskDone() reliably reachable from the failure path, so the cleanup path needs to be generation-safe before merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5674346620
Thanks for the two pointers, @SEZ9 — pulled the current head (`01a7052409e5`) directly to answer both precisely rather than from memory.
**1. Redeploy-vs-taskDone race on `cancellationFutures`.** `TaskExecutionServiceTest#testStaleTaskDoneCleansOnlyOwnedGenerationResources` (lines 573-655) is the one that exercises this. It builds an old-generation `TaskGroupExecutionTracker` and a new-generation one sharing the same `TaskGroupLocation`, seeds both `cancellationFutures` entries directly, then calls `oldTracker.taskDone(oldTask)` and asserts:
- `cancellationFutures.containsKey(oldContext)` is now `false` (old entry removed)
- `cancellationFutures.get(newContext)` is still the original `newCancellationFuture`, untouched, and it is not cancelled
- the same pattern for `asyncFutures`/`timerFlushFutures` (old cancelled, new untouched) and `executionContexts` (still points at `newContext`)
So it directly pins the exact scenario you're asking about — a stale generation's `taskDone()` firing after a newer generation has already redeployed under the same location — across all four maps, not just `executionContexts`.
**2. F1/F3 wording — the monitor claim isn't there to check anymore.** The design changed since the round where that Javadoc language existed. On the current head there is no `synchronized (TaskExecutionService.this)` block and no "same monitor as deployment" claim at all:
- `finishExecution()` (`TaskExecutionService.java:1685-1729`) does the active-context handoff via `executionContexts.compute(taskGroupLocation, (ignored, activeContext) -> ...)`, comparing `context.equals(activeContext)` inside the compute lambda — `ConcurrentMap.compute` is what serializes the transition, not a monitor. The Javadoc above it now just says "Only the active context mapping is conditional. All other resources are keyed by this tracker's context and can therefore always be cleaned safely," which matches what the code actually does.
- `deployLocalTask()`'s publication (lines 705-724) is explicit about this in its own comment: "No service-wide lock is required: publication is one-way, and cleanup uses the context's execution ID rather than TaskGroupLocation to address the exact deployment." The two-phase publish (register every resource under the new context first, then `executionContexts.put(...)` last) relies on `ConcurrentHashMap` happens-before visibility, not a shared lock.
So there's no stale monitor-vs-install mismatch to find on this head — the approach moved from lock-based to identity/compute-based between the round that Javadoc text came from and now. If you spot different wording anywhere else in the diff, a pointer would help, but I couldn't find any remaining monitor/synchronized claim tied to context installation on `01a7052409e5`.
Nothing changes about my last full review's "ready to merge" conclusion (pending your pass) — just closing out these two so you're not hunting blind.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5651854348
> two commits that aren't part of the ownership model:
>
> * [8122bdb](https://github.com/apache/seatunnel/commit/8122bdbfac22aa3a876b1812abf734ef33cbb57c) — don't close a task whose init() never ran
please rebase upstream/dev, PR #11238 merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5593914845
That works for me — deferring `CooperativeTaskWorker` is one of the two paths I explicitly said was acceptable, as long as it's tracked rather than silently dropped. Given it only bites under `task_execution_thread_share_mode: ALL/PART` (non-default), scoping it to a follow-up alongside #12164 rather than widening this PR again is the right call.
Please do file the follow-up issue and link it in the PR description (or reference it here) before merge, so it's traceable the same way #12164 is — with that in place I have no remaining blocker on the source side for this PR.
On CI: `Build` is still red on this same head (`e66911c0c6a`), but both failures are exactly the two I already flagged as unrelated — `DorisSchemaChangeIT.testDorisWithSchemaEvolutionCase` (tracked as #12131) and the `AmazondynamodbIT.testAmazondynamodb[4]` sink-visibility timing flake in `all-connectors-it-7`. Neither touches `seatunnel-engine-server`. Worth re-running just those two failed jobs rather than the whole workflow; if the DynamoDB one recurs again it's probably worth its own tracking issue, but nothing here points back to this PR's diff.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5555119917
CI status (maintainer triage, not a review of the change): the Build on 5a442e45b finished with five failures, none in this PR's diff and all tracked upstream:
- `all-connectors-it-2 (11)`: `CouchbaseIT`, KV SASL refused for longer than the writer's 30 s `waitUntilReady` (#12127).
- `all-connectors-it-4 (8)`: `AzureCosmosDBSourceIT`, the emulator answered "Service is currently unavailable" (single sighting so far, watching it).
- `transform-v2-it-part-1 (8)`: `TestFilterRowKindIT#testFilterRowKindMultiTable` on the Flink 1.18 leg, static JVM-wide counters in `AssertSinkWriter` (#12116).
- `kudu-connector-it (11)`: `KuduIT` hang after the Flink 1.18 JobManager stopped responding, cancelled at the 90-minute limit (#12132).
- `rocketmq-connector-it (8)`: broker registered under two identities after the runtime rename in `RocketMqContainer`; fixed by #12115 (green, pending merge).
Retriggered CI with an empty commit (23bbcb52); no code change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5556434375
CI status (maintainer triage, not a review of the change): the Build on 23bbcb526 finished with five failures, none in this PR's diff and all tracked upstream:
- `all-connectors-it-2 (8)`: `CouchbaseIT`, KV SASL refused for longer than the writer's 30 s `waitUntilReady` (#12127).
- `doris-connector-it (8)`: `DorisErrorIT` stack-frame race (fixed by #12114, green, pending merge) and `DorisIT#testCustomSql` reading the sink count once right after the job exited (1 row of 101 visible yet; watching it).
- `kudu-connector-it (8)` and `(11)`: `KuduIT` hang after the Flink JobManager stopped responding, both cancelled at the 90-minute limit (#12132).
- `rocketmq-connector-it (8)`: `testSourceRocketMqRestore` produced duplicate messages after the savepoint restore (#12138).
Retriggered CI with an empty commit (b555e0a9); no code change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5552672506
CI note (run still in progress, so no retrigger yet): `doris-connector-it (8)` on `a30a0bd7b` failed in `DorisErrorIT.testDoris` at `DorisErrorIT.java:85` — the pinned-frame assertion — with `DorisStreamLoad.stopLoad` → `Doris-01 stream load error` ← `UnknownHostException: doris_e2e` in stderr. Same Docker teardown race previously seen on #11503; this PR does not touch that test. Source fix: #12114 (test-only). Will post the full triage for the remaining failed jobs once this run completes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5248076175
Thanks for closing out both non-blocking items from my last review, @abdessalems.
On Issue 1: I checked the standalone run you linked, and the module-only execution is convincing given the guard's mechanical simplicity — `TaskDeployStaleContextRaceTest` passing 2/2 on the exact `e9b25d8` files (verified by your SHA-256 comparison) is good evidence. For the record on the CI-history side: the matrix run for this head (`31390599798`) is still the same `failure` conclusion I already analyzed in my last review — the `seatunnel-api`/`Dead links` failures are unrelated to this diff and the fail-fast cancellation is why no matrix leg reached `seatunnel-engine-server`, exactly as you described. Since my recommendation was "confirm the module passes" rather than "the full 82-job matrix must be green," and you've now done that independently, I'm satisfied this is closed — I wouldn't block merge on re-running the full matrix just to get a `seatunnel-engine-server` leg that would pass for reasons unrelated to this diff either way.
On Issue 2: thanks for filing #11755 and linking it here — that keeps the underlying stale-`taskDone()`/`executionContexts` race properly tracked instead of getting lost once this merges.
Both of my prior non-blocking recommendations are addressed, and my conclusion from the last full review stands: **Ready to merge**, no blockers. Nice work chasing this all the way through, including the second independent hang you caught along the way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5855122865
@davidzollo — this has approvals from @DanielLeens and yourself, and
reviewDecision reads APPROVED, so the only thing left is CI.
engine-v2-it is red on both legs from the SplitClusterFaultToleranceIT race
tracked in #12311. That PR is still open and picking up further review rounds, so
it isn't close to landing. All four unit-test legs are green here, and your own
dev merge into this branch didn't clear engine-v2-it either — which is itself
evidence the failure is dev's, not this PR's.
Would you be comfortable merging past that known dev-level flake, or would you
rather this waits for #12311?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5915852845
@davidzollo — this has approvals from @DanielLeens and yourself, and
reviewDecision reads APPROVED, so the only thing left is CI.
engine-v2-it is red on both legs from the SplitClusterFaultToleranceIT race
tracked in #12311. That PR is still open and has picked up further review rounds,
so it isn't close to landing. All four unit-test legs are green here, and your own
dev merge into this branch didn't clear engine-v2-it either — which is itself
evidence the failure is dev's, not this PR's.
Would you be comfortable merging past that known dev-level flake, or would you
rather this waits for #12311?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5628492921
@abdessalems thanks for the follow-up on f1e02f942. Taking the points in order:
**F3 / F1** — Thanks for laying out the structure. If the redeploy guard and the `deployLocalTask(...)` call both sit inside the `synchronized (this)` block opened by `deployTask(TaskGroupImmutableInformation)` as you describe, then the "same monitor" claim holds for that entry point, and I need to revisit the premise of F1 rather than just re-scope it. I'll re-read the diff on this head to confirm the enclosing block and the reworded Javadoc before marking either finding resolved; I'd rather verify against the code than close them on the description alone. If it checks out, the remaining gap is only a caller reaching `deployLocalTask` directly, and the follow-up issue you pointed at for that path is fine with me.
**F8** — The comment at `finishExecutionContext` explaining why the stale branch deliberately stays out of `finishedExecutionContexts` addresses the intent question. One small leftover from the same finding: did the redundant `get` before `remove(key, value)` also get dropped in this push, or is it still there? Either is acceptable, I just want to record the state correctly.
**F7** — Decision: keep the reflection. I don't want to widen the production API of `TaskExecutionService` purely for test access, and `ReflectionUtils.getField` on those four fields is tolerable for a race-regression test. Since the redeploy-vs-taskDone coverage naturally moves with F1/F5, please add a one-line note to the PR description that the test for that race is deferred with them.
**F1/F5** — Seen the note in the PR description; that matches what we discussed.
Remaining asks: (1) confirm the F8 `get`/`remove` status, (2) the deferred-test note for F7 in the description. I'll also want to see a green CI run on this head before signing off.
<!-- streview-comment:954 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5651359624
@SEZ9 @DanielLeens — agreed, I'll strip this back to the latch fix and carry
F1/F2/F4/F5/F6/F8 over to the #12238 discussion.
One check on scope before I rewrite the branch, so I don't have to do it twice.
Taken literally, "only the unconditional startedLatch release" would also drop
two commits that aren't part of the ownership model:
- 8122bdbfa — don't close a task whose init() never ran
- 370f175e8 — the getClassLoaders() null guard
Both were @DanielLeens's own blockers from earlier rounds, and both are required
by the latch fix itself rather than by the ownership work: moving the context
lookup inside the try block is what first makes finally — and so taskDone() and
close() — reachable on the failure path. Dropping them would reintroduce the
silent NPE that was blocked on in September.
So my plan is to keep:
b91e00582 the latch release
23011fd7d its regression test
79eebc554 the startLatchReleased rename
8122bdbfa the close() guard
370f175e8 the getClassLoaders() guard (splitting out its stale-branch leak
fix, which is ownership-related)
b69dc1161 the test-helper fix the guard needs
d98cfb608 the restored iteration count and timeout
and drop 588314d9c, 1fed68145, 912e8bd4c, e66911c0c, f1e02f942, dba310df1,
2e9d60570, e9b25d84d plus the CI-retrigger commits.
Does that match what you had in mind? It needs a force-push to rebuild the
branch on current dev, so I'd rather confirm the line first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5650209500
@abdessalems thanks for raising this before either side merges — my preference is option one: let #12238 carry the ownership model and strip this PR back to the unconditional `startedLatch` release for #11679.
The reasoning is mostly about the open findings on this thread. Nearly all of them (F1–F6, F8) are about the cherry-picked ownership model rather than the latch fix: the deploy-vs-`taskDone` race on `cancellationFutures` isn't actually closed because `deployLocalTask()` installs the context and cancellation future without the monitor `finishOwnedResources()` claims to share (and its Javadoc says otherwise), `BlockingWorker` still resolves its context from the shared `executionContexts` map instead of the tracker's `ownedContext`, the stale-cleanup branch returns without cancelling the old generation's async functions / timer flushes, the plain `put` overwrites a still-live older generation, and the heavy teardown now runs under the service-wide monitor. Fixing all of that here while #12238 re-keys the same futures by `TaskGroupContext` would mean converging two designs in the same methods, which is exactly the conflict you're worried about. Keeping #11727 to the latch fix sidest
eps that and gives #11679 a small, easy-to-land change.
Concretely, what I'd ask for:
- Drop 588314d9c, 1fed68145 and 912e8bd4c (and the follow-up work that builds on the `OwnedFuture` tagging) from this branch, leaving only the unconditional `startedLatch` release.
- Rework `TaskDeployStaleContextRaceTest` accordingly: keep whatever exercises the deploy-returns / latch contract, drop the parts that reflect into the ownership internals that will no longer exist here (F7).
- Please carry the substantive points above (F1, F2, F4, F5, F6, F8) over to the #12238 discussion so they aren't lost when the ownership model lands there.
Once the branch is trimmed I'll take another look and we can get #11679 closed independently of #12238's timing.
<!-- streview-comment:1003 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5645695055
@DanielLeens @SEZ9 — one procedural thing I noticed: both of your formal review
states are still CHANGES_REQUESTED (Daniel 09-07, SEZ9 08-28), even though the
findings from those rounds have since been addressed or deferred with agreement.
The later "no blocking issues" updates came through as COMMENTED, which doesn't
clear the earlier state, so the PR still shows as blocked on the review gate.
Could you update your review state when you get a moment if you're happy with
the current head? @SEZ9, I think this is also the final pass you mentioned once
the per-finding status was in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5412481023
@abdessalems - good question, and thanks for holding off rather than guessing. I looked into #11757 properly rather than answering from memory, since I've been reviewing it in parallel and wanted to give you a grounded answer.
Recommendation: land #11727 with the narrower change only (the `getClassLoaders()` null guard in `BlockingWorker.run()`, which you said is already written) and leave the cross-generation eviction fix entirely to #11757. Don't duplicate the ownership-check logic here.
Why: #11757 (@waterWang) already implements exactly the ownership model you're describing - `TaskGroupExecutionTracker` binds the `TaskGroupContext` it owns as a `final` field at construction, and `taskDone()`'s teardown does an identity-checked, atomic ownership-check-then-teardown under `synchronized (TaskExecutionService.this)` (the same monitor `deployTask()` already holds around `deployLocalTask()`). It's more complete than a `remove(key, value)` alone would be, since it also gates cleanup of `cancellationFutures`, async-function futures, and timer-flush futures on the same ownership check, not just the execution context. Writing the identity check again here would either duplicate that work or risk diverging from it in a way that creates a real merge conflict once #11757 lands.
#11757 is close: the design itself is confirmed sound across three independent review rounds (object-identity ownership rather than a generation counter sidesteps a whole class of off-by-one bugs). The one remaining blocker is narrow and already diagnosed: the stale/rejected branch of `finishOwnedResources()` returns without ever calling `recycleClassLoader(taskGroupLocation, ownedContext)`, so every time the guard actually fires, that generation's class loader is never released - a real leak under `classloader-cache-mode=false`, though a no-op under the default cache mode. The fix is a one-line addition before the `return` in that branch, and multiple reviewers now agree on the exact location and fix.
One correction to my own position: when I raised the eviction race as Issues 1/2 on this PR on 08-22, I should have cross-referenced #11757 explicitly instead of describing the fix in the abstract - I already had review history there by that point and didn't connect the two clearly enough, which contributed to the back-and-forth since. To be clear now: that eviction/ownership work belongs to #11757, not #11727.
So, concretely: push the `getClassLoaders()` guard here, hold the eviction change and its cross-generation test out of this PR, and once #11757's one remaining blocker lands, this PR and #11757 can merge independently in either order - #11727's guard doesn't depend on #11757's fix, and #11757's fix doesn't depend on this PR's guard being present. I'll do a fresh, focused pass on just the `getClassLoaders()` guard once you push it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5814176686
Thanks @DanielLeens, and for correcting your own earlier notes in-thread — that
kept the record straight.
So with reviewDecision at APPROVED, the only thing left is CI: engine-v2-it is
red on both legs from the SplitClusterFaultToleranceIT race tracked in #12311,
which is still open on dev. @davidzollo — is it worth waiting for #12311 to land
before merging this, or would you be comfortable merging past a known dev-level
flake given unit-test is green on all four legs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5824824562
Good, precise question — but the specific test named isn't quite right, so let me correct it before answering.
On the current head (`a80834ebf`), neither `engine-v2-it` leg is actually failing on `SplitClusterFaultToleranceIT`. I just re-pulled and grepped the raw job logs for both legs to be sure, and that class passed cleanly on the JDK 8 leg (`Tests run: 10, Failures: 0, Errors: 0, Skipped: 1`). The real failures are:
- JDK 8 and JDK 11 both fail on `BackpressureSlowSinkIT.testCheckpointsKeepCompletingUnderSustainedBackpressure` — the flake tracked by #12313/#12316, not #12311.
- JDK 11 additionally fails `CheckpointCoordinatorFailoverIT.testStreamJobFailsAfterCheckpointTriggerDispatchFailure` (a master-failover assertion mismatch) — not tracked by an issue yet as far as I can see.
#12311 (the `SplitClusterFaultToleranceIT` race) is a real, separate `dev` flake, but it isn't what's red on this PR's run. I made that same mix-up myself on 09-23 and corrected it just above this comment, so I understand how it carried over here.
None of these three failing tests touch `BlockingWorker`/`TaskExecutionService`, which is the only code this PR's diff reaches — so from a review standpoint nothing here implicates this change.
On @davidzollo's actual question: the source diff hasn't changed since `06c2ef03e` (this dev-merge is a no-op for this PR's own files), and all four `unit-test` legs are green. I don't think waiting on #12311, #12313/#12316, or the untracked `CheckpointCoordinatorFailoverIT` flake is necessary before merging — they're all `dev`-level flakes outside this diff's reach, and blocking a two-monitor-deadlock fix on unrelated E2E flakiness would just prolong the exact kind of hang this PR exists to close. Re-running the failed `engine-v2-it` jobs (no new commit needed) would likely go green on a retry and remove the ambiguity, but the merge call itself is @davidzollo's.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-6379854016
Thanks for the CI digging on head `a80834ebfb185ee0d54243e7a3ae99a5bf481817` (run 35426701856). The engine-v2-it failures (`BackpressureSlowSinkIT.testCheckpointsKeepCompletingUnderSustainedBackpressure` on jobs 105854144544 / 105854144606, and `CheckpointCoordinatorFailoverIT.testStreamJobFailsAfterCheckpointTriggerDispatchFailure` on 105854144606) exercise checkpoint completion/failure paths close to the task-lifecycle code this PR touches, so I'd rather not treat them as infra noise without a baseline run of the same scenarios on the base branch. The CDC failures on 105854144530 / 105854144535 / 105854144377 look further from this change, but the same comparison would settle that too. No causality claim either way yet — please attach the comparison with the next push.
On the review items, these still look open to me:
- **F1 / F3 / F5 (deploy vs. taskDone race):** `deployLocalTask()` still installs the context and cancellation future without taking the `TaskExecutionService.this` monitor that `finishOwnedResources()` says it shares, and does so with a plain put. Either take the monitor around the install and use a conditional insert so a still-live older generation can't be silently overwritten, or rewrite the Javadoc to describe what is actually guaranteed. The race on cancellation futures / async functions / timer flushes isn't closed until the install side participates in the same lock.
- **F2 (BlockingWorker context resolution):** `BlockingWorker` should resolve its context from the tracker's `ownedContext`, not from the shared `executionContexts` map; otherwise a reused `TaskGroupLocation` can hand it a newer generation's context/class loader. Could you confirm what that path does on this head?
- **F4 / F8 (stale-cleanup branch):** the stale branch recycles class loaders and returns, but never cancels the old generation's async functions or timer-flush tasks, and never moves `ownedContext` into `finishedExecutionContexts`. Please add both. While there, `finishExecutionContext()` can drop the redundant get before the conditional remove.
- **F6 (teardown under the service monitor):** class-loader release and async/timer cancellation now run while holding the service-wide monitor. Only the map mutations need the lock; please hoist the heavy work outside it once F1/F5 settle the ordering.
- **F7 (test coverage):** `TaskDeployStaleContextRaceTest` covers the deploy-returns contract but not the redeploy-vs-taskDone race on the cancellation futures. A test for that case would be good, ideally via package-private hooks rather than reflection into `TaskExecutionService` internals.
Once those land along with the CI baseline comparison, I'll take another full pass.
<!-- streview-comment:1999 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5601437642
@SEZ9 — summary of where each of your findings stands on the current head
(e66911c0c). Credit where it's due: most of this landed in @davidzollo's commit,
not mine.
- F2: BlockingWorker now resolves its context and class loader from the
tracker's ownedContext rather than the shared map.
- F4: the stale branch now cancels its own async-function and timer-flush
entries. They're tagged with the owning context (new OwnedFuture wrapper), so
cancelOwnedAsyncFunctionsInPlace / cancelOwnedTimerFlushTasksInPlace touch
only this generation's entries and leave a newer generation's strictly alone.
- F6: the TaskExecutionService.this monitor now covers only the map/future
bookkeeping. recycleClassLoader and all cancellations run after it's released.
- F8: the redundant get before remove(key, value) is gone. finishedExecutionContexts
is populated on the owned branch; the stale branch deliberately doesn't record
it, which @DanielLeens confirmed as the intended semantics given key reuse.
- Your Issue 2: cancellationFutures.remove(location, ownedCancellationFuture) —
self-guarding rather than invariant-dependent, as you suggested.
- F7: added testStaleTaskDoneCancelsItsOwnAsyncAndTimerFutures and
testDeployTaskIdempotentWhenAlreadyRunning as deterministic coverage, rather
than extending the stochastic race test.
- F1/F3/F5 (deployLocalTask publish/rollback and the guarded install): not in
this PR — @DanielLeens filed those as #12164 since it's pre-existing baseline
code shared with #11757, so it isn't fixed twice.
- CooperativeTaskWorker: filed as #12224, deferred with @DanielLeens's agreement
(non-default thread-share mode only).
Ready for your re-review whenever you have time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5371033903
Please feel free to leave a message on #contributors channel on the SeaTunnel Slack if you need any help in CI.
<img width="1084" height="358" alt="image" src="https://github.com/user-attachments/assets/858735bf-ce62-4bbc-a1dd-6539438b6c1b" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5860787960
Thank you for your contribution,Merge after CI passes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5882695366
Thanks for the detailed walkthrough against `a80834ebf` (same as `06c2ef03e` for the touched files). I can't mark the earlier points resolved from a description alone, so a few concrete asks before this is ready:
**Redeploy-vs-in-flight-`taskDone()` race (F1 / F5 / F7)** – I'm fine scoping that half out of this PR if the boundary is documented in code. Please point to the diff hunk in the race test that states this, and add a matching note where `deployLocalTask()` installs the new context.
**`finishOwnedResources()` Javadoc (F3)** – it still says it runs "under the same monitor used by deployment", while `deployLocalTask()` does not take that monitor around the context installation. Please either reword the Javadoc to describe the actual guarantee (per-key `executionContexts.compute(...)` ownership check) or add the monitor.
**Teardown under the service-wide monitor (F6)** – if `recycleClassLoader` / `cancelAsyncFunctionFutures` / `cancelTimerFlushFutures` now run after the `compute(...)` transition and outside any `synchronized` block, please point to that hunk in the diff so it can be verified.
**Reflection in the race test (F7)** – accepting reflection as a documented tradeoff is reasonable; please point to the diff hunk where that justification lives.
**Stale path (F4 / F8)** – your summary was cut off at item 4, so these are still open:
- Does the stale-cleanup branch cancel the old generation's async functions and timer-flush tasks after recycling class loaders? If it is intentionally deferred, please say so in a code comment.
- Should the stale path move `ownedContext` into `finishedExecutionContexts`, and has the redundant `get` before `remove(key, value)` in `finishExecutionContext()` been dropped?
**`BlockingWorker` context resolution (F2)** – not covered in your summary; does it now use the tracker's `ownedContext` rather than the shared `executionContexts` map?
Once these are answered with the corresponding diff references, I'm happy to proceed as @zhangshenghang indicated.
<!-- streview-comment:1397 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5391634898
Thanks @SEZ9 — agreed on both counts, and I want to flag one thing before we wait on a fix commit: I just re-checked the PR live and the head is still `51c6a46a` (unchanged since my last comment), so there is nothing new in the diff yet for either of us to re-review. I'll hold off on a fresh full pass until an actual fix commit lands, per the plan below.
On the two open items:
1. **Cross-generation eviction race** — agreed, `taskDone()` should capture `TaskGroupContext finishedContext = executionContexts.remove(taskGroupLocation);` (single-winner via the `ConcurrentHashMap` CAS) and recycle only that instance, rather than `recycleClassLoader()` re-resolving by key via its own `get()`. Key-only cleanup is unsafe precisely because `TaskGroupLocation` is reused across restore generations, which is the whole premise of this fix — so this needs to be closed here, not deferred.
2. **NPE / silent-null class loader path** — agreed, `taskGroupContext.getClassLoaders().get(t.getTaskID())` in `BlockingWorker.run()` needs a guard after the existing null-context check, with a fail-fast `IllegalStateException` (matching this PR's own style) instead of a silent null context class loader.
On the regression test extension: also agreed — a companion case where a newer generation installs its context under the same key before the older generation's `taskDone()` runs would pin the exact race these two issues describe, on top of the existing `TaskDeployStaleContextRaceTest` coverage.
@abdessalems — once a fix commit lands addressing both points (plus, ideally, the extended test), ping me and I'll do a fresh pass focused on just those two locations rather than re-running the full template, since the rest of the diff hasn't changed since my last approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5236032075
Thanks @DanielLeens for the detailed review and for taking the time to validate the root cause.
The `taskDone()` point is a really good catch. I was mainly focused on the
`startedLatch` path and missed that the exception before the `try` also meant
`taskDone()` was never reached.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5459113404
Thanks @SEZ9, and apologies to @abdessalems for the whiplash — I need to correct my own APPROVED review from earlier today.
I re-verified both of your blocking findings directly against the current head (`b69dc11615f0`), not just the review text, and both hold up:
**Issue 2 (BlockingWorker still reads the shared map, not `ownedContext`)** — confirmed at `TaskExecutionService.java:1086`:
```java
TaskGroupContext taskGroupContext = executionContexts.get(taskGroupLocation);
```
This PR's whole point this round was to make context resolution ownership-aware (`TaskGroupExecutionTracker.ownedContext`, introduced at `TaskExecutionService.java:1364`), but the one place that actually needs the race-free lookup — `BlockingWorker.run()` — never switched to it. It still reads through the mutable shared map, so a `deployLocalTask()` `put()` for a newer generation landing between this `get()` and the class-loader dereference at line 1094 hands the old task a new generation's context. The null-guards added in this PR pass in that scenario; they just don't guard against the case they were meant to close.
**Issue 1 (`finishOwnedResources()`'s monitor claim is one-sided)** — confirmed. The Javadoc at `TaskExecutionService.java:1494` says the teardown runs "under the same monitor used by deployment," and `finishOwnedResources()` does wrap its body in `synchronized (TaskExecutionService.this)` (line 1503). But `deployLocalTask()`'s installs at lines 592-593 —
```java
executionContexts.put(taskGroup.getTaskGroupLocation(), context);
cancellationFutures.put(taskGroup.getTaskGroupLocation(), cancellationFuture);
```
— take no lock at all. So the only actual protection is the atomic `executionContexts.remove(key, ownedContext)` inside `finishExecutionContext()` (line 1540) — and that protection is scoped to `executionContexts` alone. `cancellationFutures.remove(taskGroupLocation)` at line 1517 is unconditional and not identity-checked, so a stale tracker that loses the `executionContexts` identity check can still fall through to the "owned" branch's sibling call and delete a newer generation's cancellation future. That's the same class of bug as #11679, just moved one map over.
I missed both of these in my own from-scratch pass this morning — I traced the happy path and the specific NPE-to-hang chain from #11679 correctly, but I didn't push on whether the new `ownedContext` guarantee was actually threaded through every reader/writer, which is exactly where it's still open. Treat my APPROVED review as superseded on these two points; I concur with your CHANGES_REQUESTED and both should be blockers before merge.
On the rest of your list: agree Issues 3-6 are real (3 and 6 follow directly from the same one-sided-monitor gap; 4 and 5 are consequences of the unsynchronized `deployLocalTask()` install once fixed for 1/3 they mostly collapse together). Issue 7/8 read as reasonable low-severity follow-ups, not blockers.
One process note for @abdessalems and reviewers here: #11727's current head already carries an identity-checked `executionContexts.remove(location, ownedContext)` (via `finishExecutionContext()`), which is materially the same fix #11757 (by @waterWang, still open) proposes independently for the same removal-vs-install race. Worth the two of you coordinating so the identity-checked-removal fix lands once, in whichever PR gets there first, rather than in parallel — and #11757's PR description should probably be revisited once the gaps above (cancellationFutures, BlockingWorker's own read path) are closed here, since fixing only the `executionContexts` map leaves the same race open on the sibling maps. Separately, #11910 (the CI-refresh duplicate of this PR) is already closed, so no conflict there.
Per this round's process: no new commit has landed since my APPROVED review, so I'm not opening a new formal review right now — this comment stands in its place until a fix lands, at which point I'll do a full from-scratch re-review of the new head.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5476992296
Thanks for taking the time to re-check this against #11757's F1-F8, @SEZ9 — that's exactly the follow-up @abdessalems asked for on the 30th. I went and re-read the actual current head (`d98cfb60854b`, same SHA you cited) end to end before replying, rather than relying on my earlier pass, since your conclusion ("verification is incomplete; approval is not ready") is a real disagreement with where I'd landed. Here's what I found line-by-line.
**Issues 1, 2 and 3 (your blocking items) don't reproduce against the current head — the code already does what you say is missing.**
- Issue 1/3 claim `deployLocalTask()` installs `executionContexts`/`cancellationFutures` without taking `TaskExecutionService.this`. Checking the actual deploy path:
```java
// TaskExecutionService.java:674-678
synchronized (this) {
executionContexts.put(taskGroup.getTaskGroupLocation(), context);
cancellationFutures.put(taskGroup.getTaskGroupLocation(), cancellationFuture);
contextPublished = true;
}
```
This is inside `TaskExecutionService`, so `synchronized (this)` **is** `synchronized (TaskExecutionService.this)` — the same monitor `finishOwnedResources()` takes at line 1559. The Javadoc on `finishOwnedResources()` ("under the same monitor used by deployment", now at lines 1548-1553, not 1494 where you point) matches the code as written.
- Issue 2 claims `BlockingWorker.run()` resolves its context via `executionContexts.get(taskGroupLocation)` (your line 1086/1094). The actual current code:
```java
// TaskExecutionService.java:1136-1138
// Each tracker retains its own generation's context, so a queued worker never
// resolves a class loader from a replacement generation sharing this location.
TaskGroupContext taskGroupContext = taskGroupExecutionTracker.ownedContext;
```
It reads `tracker.ownedContext`, not the shared map — this is precisely the fix your own #11757 Issue 2 asks for, and it's already here, with a comment explicitly calling out why.
- Issue 6 (heavy teardown under the lock) doesn't hold either: `recycleClassLoader(...)` (line 1570) and `cancelAsyncFunctions`/`cancelTimerFlushTasks` (1579, 1584) all run **after** the `synchronized` block closes at line 1569 — only the ownership check and the three map removals are inside the monitor.
- Issue 8's "redundant get before remove" doesn't match current code either — `finishExecutionContext()` (1594-1600) goes straight to `executionContexts.remove(taskGroupLocation, ownedContext)` with no preceding `.get()`.
I want to flag something directly rather than let it sit implicit: every line number in your review (592, 1086, 1094, 1494, 1503, 1513-1519, 1524, 1536-1541) is off by 80-150 lines from where the equivalent logic actually sits in the current 1668-line file (674-678, 1136-1138, 1548-1614), and the code shape itself has moved past what those issues describe — this reads like the F1-F8 set was carried over from #11757's findings rather than freshly re-traced against `d98cfb60854b` as the opening line says. Given the two items marked as blocking (Issues 1 and 2) are the ones that don't reproduce, I'm not walking back my "ready to merge" conclusion on this basis — but I'd genuinely like you to re-check directly against the current head before we ask @abdessalems for another round, since the fix these describe is already in.
**Where I think you may still have a point — Issue 4.**
This is the one item whose structural description still matches current code, just at new line numbers: in the stale branch of `finishOwnedResources()` (1571-1577), `recycleClassLoader()` runs unconditionally, then the method logs and returns *before* reaching `cancelAsyncFunctions`/`cancelTimerFlushTasks`. I traced where the entries actually go, though: `taskAsyncFunctionFuture`/`timerFlushFutures` are keyed by `taskGroupLocation` with a shared inner map per location (`taskAsyncFunctionFuture.get(taskGroupLocation).put(id, future)`, line 793) — so if the new generation has already registered async functions under the same location by the time the old tracker's stale branch runs, they land in the *same* inner map object, and the new generation's own `finishOwnedResources()` will eventually remove-and-cancel that whole map, old entries included. So I don't think this is an unbounded leak the way Issue 4 phrases it — worst case it's a bounded delay (old generation's async functi
ons/timer flushes get cancelled when the new generation finishes, not immediately). Still worth a closer look together with Issue 5, but I wouldn't call either one blocking on its own.
**Issues 5 and 7** — Issue 5's "zombie tasks holding connector resources" concern doesn't materialize in practice either, since `recycleClassLoader()` always runs against the tracker's own `ownedContext` field regardless of what's currently in the shared map, so class-loader release isn't affected by the overwrite. Issue 7 (test coverage / reflection) is a fair test-quality note independent of whether the underlying race exists — happy to see a follow-up test for the `cancellationFutures` interleaving either way.
Given the above, my read stands: the two blocking items don't apply to `d98cfb60854b`, so I don't think this needs to go back to "not ready" on their account. @abdessalems — nothing needed from you on this one until @SEZ9 has a chance to confirm against the actual current line numbers; I don't want you chasing a fix for code that's already there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5473705341
Confirmed the fix landed, not just the theory, @abdessalems. I checked the actual diff between the head I fully reviewed (`410b47409b6`) and the current head (`d98cfb6085`):
```
diff --git a/.../TaskDeployStaleContextRaceTest.java
- private static final long DEPLOY_TIMEOUT_SECONDS = 10;
+ private static final long DEPLOY_TIMEOUT_SECONDS = 30;
- private static final int ITERATIONS = 5;
+ private static final int ITERATIONS = 30;
```
That's the only change anywhere in the diff - `TaskExecutionService.java` and `TaskExecutionServiceTest.java` are byte-identical to `410b47409b6`, so my last full review's conclusion carries forward unmodified rather than needing a fresh pass. This closes Issue 4 exactly as promised: `ITERATIONS` back to 30 restores the race test's regression-detection power, and restoring `DEPLOY_TIMEOUT_SECONDS` too (which I'd said was optional) is a good call for the reason you gave - a slow-but-healthy deploy on a loaded runner shouldn't be able to report a false hang. I also re-checked CI on this exact head (fork run `33297052269`): all four `unit-test` legs pass, including the restored 30-iteration `TaskDeployStaleContextRaceTest`. The two failures on this run (`all-connectors-it-2`, `doris-connector-it`) are both connector-IT jobs outside the two files this PR touches - `doris-connector-it` in particular has already failed the same way on earlier heads of this PR for unrelated reasons, as y
ou noted a few rounds back.
On your question about whether @SEZ9's Issues 1 and 6 on #11757 apply here: yes, and I traced the current code to confirm exactly what you described. `finishOwnedResources()` (`TaskExecutionService.java:1555`) calls `recycleClassLoader(taskGroupLocation, ownedContext)` unconditionally at line 1570, before the `!contextOwned` check - so in the stale branch, this tracker's own class loader gets recycled regardless of whether its own async-function/timer-flush entries (registered under the same reused `taskGroupLocation`) are still live and potentially still running against that class loader. That's the same underlying gap as Issue 2 in my last full review (carried from @SEZ9's Issue 4 on #11757: the stale branch never cancels its own async-function/timer-flush entries), just described from the other direction - not a new, independent defect.
I checked how much this actually bites in practice: `DefaultClassLoaderService.releaseClassLoader()` (`seatunnel-engine-core/.../DefaultClassLoaderService.java:102-121`) decrements the reference count and returns immediately when `cacheMode` is true (`ServerConfigOptions.CLASSLOADER_CACHE_MODE` defaults to `true`) - it never removes the class loader from `classLoaderMap` or calls `recycleClassLoaderFromThread()` in that branch. So under the default configuration, an orphaned async/timer task racing this ordering keeps working off the still-live, still-cached class loader; nothing gets pulled out from under it. The real exposure is scoped to `classloader-cache-mode=false`, and only when the reference count from this call happens to hit exactly zero - the same conditional severity that's already been established for the sibling issues in this thread.
Given that, I don't think this needs to be a new blocker on top of what's already tracked, and I don't think it needs fixing twice. My preference: leave it exactly where Issue 2 already sits in my last review - a Medium, non-blocking follow-up - and let it land in whichever of #11727 / #11757 gets there first, same as the coordination approach we already settled on for the ownership-model work itself. If you'd rather close it here since `finishOwnedResources()` already lives in this PR, that's fine too; either way, please don't block on my account.
Net: no change to my conclusion. **Ready to merge** - the one open ask from my last review (the iteration-count restore) is done and verified in the diff, CI is green on everything this PR actually touches, and the remaining Issues 1-3 stay as accepted non-blocking follow-ups. As before, I'm a comment-only reviewer here, so a maintainer with write access (@davidzollo / @zhangshenghang) would still need to do the actual approve/merge step.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5509606968
**CI diagnosis and status update (head `d98cfb60854b` → `6770a68d239d`)**
I dereferenced the apache-side `Build` check pointer to the fork's real run (`abdessalems/seatunnel` run `33297052269`, head `d98cfb60854b`) and pulled the failing job logs directly:
- All `unit-test` legs (JDK 8/11, Linux/Windows) and both `engine-v2-it` legs — the suites that actually exercise `TaskExecutionService`/Zeta worker-startup, the only production file this PR touches — passed.
- `all-connectors-it-2 (8, ubuntu-latest)` failed on `CouchbaseIT.testFakeSourceToCouchbaseSink` (Couchbase test-container bootstrap failure), a known dev-wide infra flake unrelated to this PR's connector-doris/engine scope.
- `doris-connector-it (11, ubuntu-latest)` failed on `DorisErrorIT.testDoris:85` (`expected: <true> but was: <false>`), a Doris-connector-specific assertion with no relationship to `TaskExecutionService` or the latch-release fix.
- `all-connectors-it-6 (11, ubuntu-latest)` was cancelled (its JDK 8 counterpart passed).
None of these three legs touch `seatunnel-engine-server` or any file changed by this PR. This matches @DanielLeens's own 09-01 review, which reached the same conclusion independently.
I don't have write/admin access to `abdessalems/seatunnel` (fork collaborator check returns pull-only), so I can't trigger a job-level or run-level rerun directly. Since this PR's history already has precedent for using an empty commit to retrigger CI when a direct rerun isn't available (`17a98e1e1d`), I pushed an equivalent empty commit (`6770a68d239d`) via the maintainer-can-modify permission to get a fresh `Build` run started; no production or test code was touched.
**Review status**: I read @DanielLeens's 09-01 full re-review and @SEZ9's 08-31 review end to end. Daniel's 09-01 pass already independently re-traced every one of SEZ9's F1–F8 findings against the current file and concluded "Ready to merge" with zero blocking items — the two non-blocking follow-ups (F5/Issue 1: a pre-existing redeploy-vs-cancel gap that predates this PR and is out of its stated scope; F7/Issue 2: a cosmetic reflection-based field access in the race test) are explicitly marked non-blocking and out of scope, so I made no source changes on their account. There are no unresolved inline review threads on this PR (all review feedback was posted as top-level review bodies).
Net: no source or test changes from me on this pass — this PR's remaining gate is CI (now re-running) plus maintainer approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5645683255
CI finished on ce142f830. The dev sync did what @DanielLeens expected:
rocketmq-connector-it is green on both legs now. All four unit-test legs pass
and seatunnel-engine-server is clean — 69 success, 14 skipped, 10 failures.
The 10 are all connector/E2E: all-connectors-it-7 (Testcontainers/MinIO
startup), all-connectors-it-6 and -it-2 (Docker registry pull), engine-v2-it
(which @DanielLeens showed fails identically on dev's own code at 6ee0c374478),
and paimon-connector-it (8 and 11), which is new on this run and I haven't
traced yet.
Nothing touches this PR's diff. @SEZ9 — this is as clean a run as I can get from
my side; happy to rerun any of them if you'd like to see a specific one pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5644359192
@SEZ9 per-finding status on the current head:
F1 — deferred. The publish/rollback race in deployLocalTask() is tracked in
#12164; not addressed here.
F3 — addressed. The Javadoc now describes what is actually guaranteed:
deployTask(TaskGroupImmutableInformation) opens synchronized (this) at line 528
and the deployLocalTask(...) call at line 550 sits inside it, so that entry point
is covered; a caller reaching deployLocalTask directly is not, and the text says
so and points at #12164.
F2 — addressed. BlockingWorker resolves its context and class loader from the
tracker's ownedContext, not the shared executionContexts map.
F4 — addressed. The stale path cancels the old generation's async-function and
timer-flush entries. They carry the owning context (OwnedFuture wrapper), so
cancelOwnedAsyncFunctionsInPlace / cancelOwnedTimerFlushTasksInPlace touch only
this generation's entries and leave a newer generation's untouched.
F5 — deferred with F1 to #12164 (same plain-put install path).
F6 — addressed. The TaskExecutionService.this monitor now covers only the
ownership check and the map/future bookkeeping; recycleClassLoader and all
cancellations run after it is released.
F8 — addressed. The redundant get is gone: finishExecutionContext goes straight
to executionContexts.remove(taskGroupLocation, ownedContext) at line 1724. The
stale path deliberately does not move ownedContext into finishedExecutionContexts,
and there is now a comment there saying why — TaskGroupLocation is reused across
generations, so filing it under a key a newer generation owns would make later
lookups resolve the wrong context.
F7 — noted in the PR description as deferred.
On CI: @DanielLeens was right about the cause. I've merged the latest dev into
the branch — RocketMqIT now has the waitForTopicRoute() calls that close the
broker-route race, which this branch was missing. Rerunning CI on the new head.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5652234280
Rebased onto dev now that #12238 has merged, and stripped this back to the
original scope as @DanielLeens and @SEZ9 suggested.
What's left is 3 files, +370/-11: the unconditional startedLatch release for
#11679, its regression test, and the two guards that the latch fix itself makes
necessary — a missing per-task class loader now fails fast instead of installing
a null context class loader, and close() only runs on a task whose init() was
attempted.
Dropped: the three cherry-picked ownership commits from #11757 (588314d9c,
1fed68145, 912e8bd4c), the OwnedFuture work built on them, and the ownership
Javadoc — all superseded by #12238. The merge conflict with dev is gone as a
result.
@SEZ9 — F1, F2, F4, F5, F6 and F8 were all about that ownership model, so as you
asked they belong on the #12238 discussion rather than here. F7's reflection
concern no longer applies, since the test no longer reaches into the ownership
internals.
Locally green: 21 tests, 0 failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5812700965
@SEZ9 — thanks for pushing back rather than taking my last comment at face value; two corrections are fair, so let me be precise this time.
**Aggregate review/merge state (re-confirmed just now via the API, not the sidebar):** `reviewDecision` = `APPROVED`, `mergeable` = `MERGEABLE`, `mergeStateStatus` = `BLOCKED`. The only failing required check is `Build`. So nothing about review state (mine, yours, or anyone else's) is gating this merge right now — it's purely the `Build` check.
**F4 — my mistake, it was dropped from the list, not resolved.** On the current head, `finishExecution()` (`TaskExecutionService.java:1685-1730`) already does exactly what F4 asked for: it unconditionally calls `cancelAsyncFunctionFutures(context)` (`:1720`) and `cancelTimerFlushFutures(context)` (`:1725`) for the tracker that's finishing, regardless of whether `activeGeneration` is true — i.e. a stale generation's own async functions and timer-flush tasks are cancelled too. But this method is untouched by this PR: it's part of `dev`'s existing design from #12238 (already at the merge-base), not something this PR's 3-hunk diff changes. So F4 is satisfied on `dev`, out of scope for this PR — same bucket as F1/F3/F5/F6/F8, I just missed naming it explicitly.
**F2 and F7 aren't still open** — I want to be direct about this since "still pending verification" suggests otherwise. Both were confirmed with line numbers in my full review on `06c2ef03e` (https://github.com/apache/seatunnel/pull/11727#pullrequestreview-5254567038, "Response to the open @SEZ9 items" section), and I independently re-diffed and re-confirmed them again yesterday on `a80834ebf` (both files are byte-identical to `06c2ef03e`, so the same evidence applies unchanged):
- F2: `BlockingWorker.run()` reads the class loader from `taskGroupExecutionTracker.context.getClassLoaders()` (`:1294` in that review's line numbering), not the shared `executionContexts` map. `context` is `private final`, assigned once in the constructor from the same object `deployLocalTask()` builds — a newer generation's context can't leak in.
- F7: the only reflective access is `executionContextsOf()` in `TaskDeployStaleContextRaceTest.java` (now with a class-level Javadoc stating what it does and doesn't cover), plus the same Javadoc scoping the redeploy-vs-taskDone race to `deployLocalTask()`/tracker teardown — which is #12238/dev territory, not this diff.
**F1/F3/F5/F6/F8 — direct answer to "introduced by this PR or only present on dev":** only present on `dev`, not introduced or touched by this PR. Evidence: `git grep -n "finishOwnedResources\|cancelOwnedAsyncFunctionsInPlace\|ownedContext" <head>` returns zero hits — those identifiers don't exist anywhere in the current tree, PR or dev; they were from the earlier "ownership model" commits that were stripped out on 09-13 per the discussion above. And the file-scoped diff (`git diff <merge-base> <head> -- TaskExecutionService.java`) is exactly: one `import java.io.IOException;` removal and two hunks inside `BlockingWorker.run()` (`:1276-1356`) — nothing in `deployLocalTask()`, `finishExecution()`, or `cancelAllTask()` is touched. So F1 (redeploy-vs-taskDone race on `cancellationFutures`), F3, F5, F6 and F8 all describe `dev`/#12238 code this PR doesn't reach.
**CI on `a80834ebf` — correcting my own 09-23 comment, which named the wrong test.** I pulled the actual fork job logs rather than go from memory again:
- `engine-v2-it (8, ubuntu-latest)` and `engine-v2-it (11, ubuntu-latest)`: **both** fail on `BackpressureSlowSinkIT.testCheckpointsKeepCompletingUnderSustainedBackpressure` — not `SplitClusterFaultToleranceIT` as I said on 09-23, that was wrong and I apologize for the confusion it may have caused. This is the same known `dev` flake I flagged in my `06c2ef03e` review (fixes #12316/#12313 still open upstream).
- `engine-v2-it (11, ubuntu-latest)` additionally fails `CheckpointCoordinatorFailoverIT.testStreamJobFailsAfterCheckpointTriggerDispatchFailure` — an assertion mismatch during simulated master failover (`can't find task group address from taskGroupLocation` inside `CheckpointCoordinator.notifyCheckpointCompleted`), unrelated to `BlockingWorker`/`TaskExecutionService` and not something I'd seen on earlier heads of this PR.
- `all-connectors-it-2 (8 and 11, ubuntu-latest)`: `OpengaussCDCIT.testAddFieldWithRestore` — Testcontainers/Awaitility timeout plus a "container is not running" Docker error mid-test.
- `all-connectors-it-7 (11, ubuntu-latest)`: `PostgresCDCIT.testPostgresCdcSnapshotOnlyAndCommittedOffsetStartupModes` — `awaitility` timeout after a replication-connection reset.
- `kudu-connector-it (8 and 11, ubuntu-latest)`: cancelled, not investigated.
None of these touch `seatunnel-engine-server`'s `TaskExecutionService`/`BlockingWorker` path, so none implicate this diff. 72 of 79 non-skipped jobs pass, including all four `unit-test` legs.
I've done a complete, from-scratch re-review of the current head and I'm posting it as a new formal review right after this, since a new commit (the dev-merge to `a80834ebf`) landed since my last one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5673570163
@abdessalems thanks for the heads-up — no surprise on my side, and the F7 reflection point is settled as far as I'm concerned. Keeping the single reflective access into `executionContexts` (line 276) so the remover thread can pull an entry mid-deploy is exactly the trade-off I asked for: a narrow, test-setup-only hook rather than widening the production API for test visibility. The fact that none of the ownership-internals reflection remains and that the `TaskExecutionServiceTest` additions are reflection-free is what I wanted to hear.
Acknowledged that the test Javadoc update landed in 01a705240 and the PR description was refreshed the same day; I'll pick those up in my pass.
Two small asks to close out the review scope:
1. **F7, coverage half** — the original finding also noted the race test covered the deploy-returns contract but not the redeploy-vs-taskDone race on `cancellationFutures`. Could you point me at which of the new `TaskExecutionServiceTest` additions (or which case in `TaskDeployStaleContextRaceTest`) exercises that path, so I can verify it directly rather than infer it?
2. **F1/F3 wording** — when I re-read `finishOwnedResources()` I'll be checking that the Javadoc's monitor claim now matches what `deployLocalTask()` actually does around context installation. If you've already adjusted either the code or the comment for that, a one-line pointer saves me a hunt.
I'll go through the remaining files on the current head and follow up here.
<!-- streview-comment:1046 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5690933952
Thanks for the scope correction, @abdessalems — that's a fair point, and I'll go with it.
Given that `finishOwnedResources()` and `testStaleTaskDoneCleansOnlyOwnedGenerationResources` are not in this PR's diff, I'm treating F1, F3 (the monitor/Javadoc claims) and F5 (the plain-put overwrite in `deployLocalTask`) as out of scope for #11727 and will follow them up against the change already on `dev` rather than block here. Same for the coverage half of F7 — the redeploy-vs-taskDone test on `cancellationFutures` living elsewhere is fine by me.
What I still need from you on this PR's own diff:
1. **F2** — since `BlockingWorker.run()` is the core change here, please confirm (or point me at the lines) that it now resolves the context from the tracker's `ownedContext` rather than the shared `executionContexts` map, so a reused `TaskGroupLocation` can't pick up a newer generation's class loader.
2. **F7 (reflection part)** — `TaskDeployStaleContextRaceTest` still reaches into `TaskExecutionService` internals via reflection. If there's a way to drive the deploy-returns contract through a package-private hook or the existing test helper instead, I'd prefer that; if not, a short comment in the test explaining why reflection is required is enough.
3. **F4 / F6 / F8** — please confirm whether the stale-cleanup branch, the teardown-under-monitor path, and `finishExecutionContext()` are touched by this diff at all. If they aren't, I'll move those to the same follow-up as F1/F3/F5 and they won't hold this PR.
Once I have those three answers I can do my final pass.
<!-- streview-comment:1072 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5696970801
CI update on 2dc214681, @DanielLeens — re-ran the failed jobs. All four
unit-test legs pass now, including unit-test (8, ubuntu-latest), which had failed
on the first attempt.
engine-v2-it failed on both legs on the rerun, though. I can't read the job logs,
so I'd rather not call it a flake on my own: on 01a705240 it failed on JDK 8 and
passed on 11, and on ce142f830 it failed on both, which is when you traced it to
the same SplitClusterFaultToleranceIT failure on dev's own code.
Since you've been pulling these logs directly — would you mind checking whether
this is still that same upstream failure, or something new? The other reds are
the usual all-connectors-it-2/6/7 and paimon-connector-it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5862310940
Thanks for the follow-up, @abdessalems.
On the review side: the automated summary above says the three items I flagged earlier — the deploy/teardown monitor (finishOwnedResources() vs. deployLocalTask()), the `BlockingWorker` context source, and the stale-cleanup path — were addressed at `06c2ef03e`, and that the current head `a80834ebf` is byte-identical to `06c2ef03e` for the files this PR touches. I'm happy to take that as the working state, but I'd like to close the loop on the rest of that review before we talk about merging. Could you confirm, ideally with a short pointer to the relevant change in `TaskExecutionService.java` / `TaskDeployStaleContextRaceTest.java`, how each of these landed (or why you consider them not applicable):
- the plain `put` in `deployLocalTask` overwriting a still-live older generation's context/cancellation future;
- heavy teardown (class-loader release, async-function and timer-flush cancellation) running under the service-wide `TaskExecutionService.this` monitor;
- the race test covering the redeploy-vs-taskDone race on `cancellationFutures` rather than only the deploy-returns contract, and whether the reflection into `TaskExecutionService` internals can be avoided;
- the stale path not moving `ownedContext` into `finishedExecutionContexts`, and the redundant `get` before `remove(key, value)` in `finishExecutionContext()`.
A one-line-per-item response is fine if they're already in the diff; I just don't want to rely on a summary that only lists three of them.
On CI: I'll go with what @zhangshenghang said — we merge once CI passes. I understand your point that the engine-v2-it failure looks like the race tracked in #12311 rather than something this PR introduced, but I'd rather not merge past a red leg on a change that is itself about a deploy/teardown race. If #12311 stays open for a while, a re-run once it (or a fix for that flake) lands is the cleanest path; if you have evidence the failure is unrelated beyond the dev-merge observation, feel free to post it here and we can reassess.
<!-- streview-comment:1365 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5923571862
Thanks for the update, @abdessalems. Before we talk about merging past the integration-test failure you describe, I need to close the loop on the review findings from the last round — I don't see them addressed in this thread yet, and nothing in your comment speaks to them.
Concretely, these are still open from my side:
1. **Redeploy-vs-taskDone race (PR11727-F1 / F3 / F5)** — `finishOwnedResources()` documents that it runs under the same monitor as deployment, but `deployLocalTask()` installs the context and cancellation future into the shared maps via a plain `put` without taking that monitor. That means the race on `cancellationFutures`, async functions and timer flushes is not actually closed, and the Javadoc is currently inaccurate. Please either take the `TaskExecutionService.this` monitor around the installation (or otherwise make the install/finish pair atomic) and make the `put` refuse to silently overwrite a still-live older generation, or update the Javadoc to describe what's really guaranteed.
2. **`BlockingWorker` context lookup (PR11727-F2)** — it still resolves its context from the shared `executionContexts` map rather than the tracker's `ownedContext`, so a worker can pick up a newer generation's context/class loader for the same reused `TaskGroupLocation`. Please route it through `ownedContext`.
3. **Stale-cleanup branch (PR11727-F4 / F8)** — after recycling the class loaders it returns without cancelling the old generation's async functions or timer-flush tasks, and it never moves `ownedContext` into `finishedExecutionContexts`. The redundant `get` before `remove(key, value)` in `finishExecutionContext()` can go at the same time.
4. **Teardown under the service-wide monitor (PR11727-F6)** — class-loader release and async-function/timer cancellation now run while holding `TaskExecutionService.this`. If the monitor is kept for the map mutations, please move the heavy teardown outside it.
5. **Test coverage (PR11727-F7)** — `TaskDeployStaleContextRaceTest` covers the deploy-returns contract but not the `cancellationFutures` race above, and reaches into internals via reflection. Once (1) is fixed, a test that exercises the redeploy-vs-taskDone path directly would be ideal.
On the CI question itself: I can't verify from this thread which legs are red or why, and a flaky integration test on dev doesn't change the fact that the items above are correctness issues in this change. Once they're addressed (or you tell me why a given one doesn't apply), I'm happy to look at the CI situation with you and decide whether a dev-level flake should block.
Could you push a revision covering the points above, or reply inline on any you disagree with?
<!-- streview-comment:1439 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5411577006
@DanielLeens
The part that actually prevents that is the remove(key, value) identity check against the context pinned at deploy time, which needs the tracker to hold the context it owns. That looks like the same ownership model as ownedContext in #11757, so rather than write it twice or risk conflicting logic with @waterWang's work, I'd rather confirm the intended split with you.
Would you prefer the ownership fix here in #11727, with #11757 rebasing on top? Or should #11727 land with the narrower change and leave the identity check to #11757? Happy to go either way, I just didn't want to guess.
I'll hold the eviction change and the cross-generation test until you've had a chance to weigh in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5564946444
Filed Blocker 1 (the context-publish leak / permanent-redeploy-block) as apache/seatunnel#12164, since it's pre-existing baseline code shared with #11757's branch too, not specific to this PR - tracking it there so the fix (wherever it lands) isn't duplicated across both PRs. Blocker 2 (stale generation's own async/timer futures not cancelled) remains specific to this PR's own diff and stands as-is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5558417751
CI note (maintainer triage): the Build on 18e821c3d has already lost `all-connectors-it-5 (11)` (`FtpFileIT#testFtpBinaryUpdateModeContinuousDiscoveryWithNonRecursiveScan`: job reached CANCELED but a `checkpoint-coordinator-1/<jobId>` thread stayed alive; second sighting after #11757, filed as #12147) and `all-connectors-it-7 (11)` (`PostgresCDCIT` committed-offset restore, #11864). Neither is in this PR's diff. I will retrigger once this run finishes rather than cancel a running build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5578009916
Thanks @DanielLeens for the self-correction and for tracing this end-to-end — agreed with the reopen.
**Blocker 1 (context-publish failure leaks a redeploy-blocking entry):** I agree with your read of `deployLocalTask`: once `executionContexts.put` / `cancellationFutures.put` run under `synchronized (this)` and `contextPublished` flips to `true`, a throw from `onContextPublished.run()` or either submit call bypasses `onFailureBeforeContextPublished`, so the entry stays behind. Since you note this is pre-existing baseline code shared with another branch, I'm fine tracking the fix in the separate issue you filed rather than blocking this PR on it. Two things on that: (1) your review body appears cut off mid-sentence after "either submit call" — could you paste the rest so the record here is complete? (2) Please confirm whether any of this PR's ownership changes (the `ownedContext` / `finishOwnedResources()` path) depend on that leaked entry being absent; if so we should at least note that dependency here so the follow-up doesn't regress it.
**Blocker 2 (stale generation's own async/timer futures not cancelled):** this is the same as my earlier F4 (stale-cleanup branch recycles class loaders but never cancels the old generation's async functions or timer-flush tasks). It's specific to this diff and remains a blocker for this PR.
Since you've confirmed the head is unchanged from `370f175e879` through `18e821c3d`, the rest of my earlier findings are also still open as far as I can tell:
- F1/F3: `finishOwnedResources()` javadoc says it runs under the deployment monitor, but the context/cancellation-future installation in `deployLocalTask` doesn't actually close the redeploy-vs-taskDone race on `cancellationFutures`; either take the monitor there or fix the javadoc and the race separately.
- F2: `BlockingWorker` should resolve its context from the tracker's `ownedContext`, not the shared `executionContexts` map, to avoid picking up a newer generation for a reused `TaskGroupLocation`.
- F5: the plain `put` overwrites a still-live older generation's entry/future silently.
- F6: heavy teardown (class-loader release, async/timer cancellation) under the service-wide monitor.
- F7: `TaskDeployStaleContextRaceTest` doesn't exercise the `cancellationFutures` race and relies on reflection into internals.
- F8: stale path never moves `ownedContext` into `finishedExecutionContexts`; redundant `get` before `remove(key, value)` in `finishExecutionContext()`.
Concrete asks before I can re-approve: a new head that addresses Blocker 2/F4, F1/F3, F2 and F5 (F6–F8 can be follow-ups if you'd prefer, just say so), plus the completed text of the Blocker 1 write-up.
<!-- streview-comment:875 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5611965989
@abdessalems thanks for the point-by-point status on e66911c0c — that's exactly what I needed, and apologies for the delayed answer.
Where I land on each:
- **F2 / F4 / F6 / Issue 2** — the described changes (ownedContext for BlockingWorker, the OwnedFuture tagging so the stale branch only cancels its own generation's async-function and timer-flush entries, teardown moved outside the `TaskExecutionService.this` monitor, and the self-guarding `cancellationFutures.remove(location, ownedCancellationFuture)`) address what I raised. I'll confirm against the diff on re-review, but nothing further to ask on these.
- **F8** — fine with the redundant `get` removed and with the stale branch deliberately not recording into `finishedExecutionContexts`; a one-line comment at that branch stating it's intentional (because of key reuse) would save the next reader the same question.
- **F7** — `testStaleTaskDoneCancelsItsOwnAsyncAndTimerFutures` and `testDeployTaskIdempotentWhenAlreadyRunning` as deterministic coverage is the better approach. One question: do the new tests still reach into `TaskExecutionService` internals via reflection, or did you get a non-reflective seam for them?
- **F1 / F5** — accepting the deferral to #12164 since it's shared baseline code. Please add a short note in the PR description that the redeploy-vs-taskDone publish race is tracked there, so nobody reads this PR as closing it.
- **F3** — this one I'd still like handled here rather than in #12164: the `finishOwnedResources()` Javadoc is text introduced by this PR, and as long as `deployLocalTask()` doesn't take that monitor, the "under the same monitor used by deployment" wording is inaccurate. Please reword it to describe what the monitor actually covers now (the map/future bookkeeping), and reference the follow-up for the deployment side.
So the remaining asks are: the F3 Javadoc fix, the F8 comment, the F7 reflection answer, and the description note for F1/F5. Once the F3 wording is pushed I'll do the re-review.
<!-- streview-comment:942 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5662012394
Thanks for the correction, @abdessalems — that matches what I found independently in my own re-review pass on this same head. In `TaskDeployStaleContextRaceTest.java`, the reflective access into `executionContexts` (around line 276, used so the remover thread can simulate an entry disappearing mid-deploy) is the one field access via reflection I noted; it's scoped to test setup only and is not reaching into any of the ownership-model internals the earlier rounds of this PR used reflection for. I called that out as narrow and benign in my last full review, consistent with @SEZ9's earlier call to keep it rather than widen the production API for test-only visibility — so no new concern from my side on F7.
For the record, my last full review (against this same current head, `01a7052409e`) found no open blockers: the class-loader resolution now reads from `taskGroupExecutionTracker.context` (pinned at construction, before the tracker's context is ever published into the shared map), the `close()`/`initAttempted` guard and the exactly-once latch release are both independently correct, and the two issues from my prior round (duplicated `getActiveExecutionContext` logic, and stale framing in the PR description/test Javadoc about the pre-#12238 trigger) are both closed. I'm at "ready to merge" from a source standpoint, pending @SEZ9's pass on the remaining files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5357532819
结论:我会继续协助把当前 head 的异常 CI 修到可交付状态。
我刚重新核了一遍当前运行,`Build` 的失败点落在 `unit-test (8, windows-latest)`,我这边会先继续确认这是当前分支代码引起的确定性问题,还是旧测试/分支漂移带来的非目标失败;如果是可以在这个分支上做的最小安全修复,我会直接补丁并回推。
这一步你这边暂时不用再做额外操作,我先往下处理。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5357642598
结论:我已经把可写分支上的替代 PR 建好了,后续我会继续在那个 PR 上跟进当前 head 的 CI。
我刚实际尝试把最新 `dev` 合回这个 fork 分支并以 maintainer 身份回推,但对 `abdessalems/seatunnel` 的 push 仍然返回 `403`,所以这个 PR 不能原位刷新到一个新的可跑 head。
为了不让这个 fix 卡在 fork 权限上,我把当前 head(`45a6d3bf709fc8f35ee40911c6c491cf0f54ac2e`)合并最新 `dev` 后发到了可写分支,替代 PR 在这里:#11910
如果 #11910 的 CI 先恢复正常,后续建议直接在那个 PR 上继续 review / merge;功能修复本身没有额外新增,只是换到了一个可维护的 head 上继续跑检查。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5788006108
Noting the dev sync: since `a80834ebf` carries the same patch content as `06c2ef03e`, the merge itself doesn't need a fresh pass from me. What I still need is a pointer for each of the earlier `TaskExecutionService` findings, since I can't tell from this thread which of them landed on the current head:
- **PR11727-F1 / F3 / F5** – `deployLocalTask()` installs the context and cancellation future with a plain `put` outside the `TaskExecutionService.this` monitor that `finishOwnedResources()` claims to share, so the redeploy-vs-taskDone race on `cancellationFutures` / async functions / timer flushes isn't actually closed, and a still-live older generation can be silently overwritten. Please either take the monitor around the installation (and reject/handle a live older entry instead of overwriting it) and align the Javadoc, or point me to where this was changed.
- **PR11727-F2** – `BlockingWorker` should resolve its context from the tracker's `ownedContext` rather than the shared `executionContexts` map, so it can't pick up a newer generation's context/class loader for a reused `TaskGroupLocation`.
- **PR11727-F4 / F8** – the stale-cleanup branch should also cancel the old generation's async functions and timer-flush tasks and move `ownedContext` into `finishedExecutionContexts`; the redundant `get` before `remove(key, value)` in `finishExecutionContext()` can be dropped.
- **PR11727-F6** – if the class-loader release and cancellation work stays under the service-wide monitor, a short note on why that's acceptable would help; otherwise move the heavy teardown outside the lock.
- **PR11727-F7** – a test exercising the redeploy-vs-taskDone race on `cancellationFutures` in `TaskDeployStaleContextRaceTest` would round this out, ideally with less reflection into internals.
If some of these are already in `06c2ef03e`, a one-line reply per item with where to look is enough and I'll verify against `a80834ebf`.
<!-- streview-comment:1247 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5385709929
Flagging for the record: a new commit landed after my last comment — `51c6a46a` ("Merge branch 'dev' into fix/11679-blockingworker-start-latch", 2026-08-22T19:00), on top of the `72268def` head my Issues 1 and 2 above were written against. That's technically a new-commit trigger for a fresh full re-review, so I want to be explicit about that rather than quietly treating it as covered.
That said, I diffed the two heads before writing this: `git diff 72268def..51c6a46a -- TaskExecutionService.java TaskDeployStaleContextRaceTest.java` is empty — the merge pulls in unrelated `dev` changes (docs, workflows, other connectors) with zero net change to either file this fix touches. So the code my Issues 1 and 2 describe is byte-identical at the new head, and both remain open:
1. **Cross-generation eviction race (High)** — `taskDone()`'s key-only `executionContexts.remove(taskGroupLocation)` and `recycleClassLoader()`'s independent re-`get()` can still let a stale generation's cleanup evict and release a newer, live generation's context and class loaders, since `TaskGroupLocation` is reused verbatim across restore generations by design.
2. **Residual NPE / silent-null path (Medium)** — `getClassLoaders().get(t.getTaskID())` still isn't guarded against a concurrent `setClassLoaders(null)` from the race in (1), and a missing per-task entry installs a silent null context class loader instead of failing fast.
@SEZ9's from-scratch review at this same head (`51c6a46ae449`) independently reconfirms both at the same locations (`TaskExecutionService.java:1430` / `:1481-1494` for the eviction race, `:1096` for the NPE/silent-null path), with the same suggested direction I'd already converged on: make cleanup ownership-aware (pin the specific `TaskGroupContext` at deploy time, atomic `remove(key, value)`, skip recycle on identity mismatch) rather than key-only.
So: no new findings from me this round, but also no change to my position — these two are in the same method this PR touches and the same race family the fix targets, and I'd still like to see them closed here rather than deferred. Given the diff is a no-op on the reviewable code, I'm not re-running the full review template again; happy to do a fresh pass once an actual fix commit lands for either issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5240680592
Thanks Daniel. Added the test for your Issue 3 in e9b25d8.
I verified it red/green before pushing, on the JDK 8 leg, same commit, with the
test file unchanged between the two runs:
- with the `taskDone()` guard reverted, the new method fails on iteration 0 -
*"The task group future did not complete within 30s"* - while the existing
deploy-returns method still passes. That is exactly the blind spot you
described: the old assertion is already satisfied when `startedLatch` is
released, which is before `taskDone()` runs.
- with the guard restored, both methods pass, 30 iterations in 4.5s.
The new test calls `deployLocalTask()` directly so it can hold the
`TaskExecutionState` future that `deployTask(Data)` discards, and it keeps the
remover thread racing until after that future has been observed - otherwise the
window closes before `taskDone()` is reached. Completing exceptionally counts as
completing; only never settling fails it.
Still to come: the follow-up issue for the underlying stale-`taskDone()` /
`executionContexts` race. I will link it here and on #11679 once it is filed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5233641538
Thanks for your contribution! This is a solid addition to the project, and your attention to detail really shows. Contributors like you are what keep this community moving forward. Feel free to keep the PRs coming — we're always glad to have you here!
Please ignore the following message if you have joined in the Slack~
```
To facilitate focused communication among contributors, we’ve created a brand-new dedicated #contributors channel in official Slack workspace.
Slack workspace link: https://s.apache.org/seatunnel-slack
This channel is designed for technical discussions, feature design brainstorming, roadmap sync and direct communication with project core Contributors. Everyone who contributes code, docs, test cases or community ideas is warmly invited to join, share thoughts and collaborate together under The Apache Way. Welcome to join 😄
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5265834604
Nothing outstanding from my side. The current head (`79eebc5`) is the exact commit I reviewed and approved — a pure rename with zero behavior change on top of the already-verified fix. To recap: the latch/context release is unconditional on every exit path, idempotent by construction, backed by two non-flaky regression tests (deploy-hang and task-group-completion-hang), and all four unit-test legs are green on this head including the new `seatunnel-engine-server` coverage. No blockers.
(For anyone following the #11757 coordination note in my last review: that's about sequencing *that* PR's rebase after this one, not something #11727 needs to wait on.)
Ready to merge whenever a committer has a chance, @zhangshenghang @davidzollo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5557010785
Cross-link: the `doris-connector-it (8)` failures on this PR's recent runs (Doris container `Timed out waiting for container port to open` after the 60 s default, and `DorisIT` reading an empty / partial sink table right after the job exited on the Spark legs) are addressed at the source by #12142 (test-only: six-minute startup budget for the all-in-one image, bounded visibility wait for the 2pc sink checks). Nothing to do on this PR; it will be retriggered once its current Build finishes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5707337983
Thanks @abdessalems for checking each point against the diff.
1. **F2** — The change you describe (the worker reading the class loader from the tracker's pinned context, with the shared-map lookup and null-context branch removed) is exactly what F2 asked for. I'll confirm it against the diff before marking it resolved.
2. **F7 (reflection)** — Understood. If the only non-reflective paths to the map tear the task group down rather than race the deploy, a package-private accessor would be production surface for test-only use, which I don't want. Keeping the reflection with the Javadoc on `executionContextsOf()` in 2dc214681 is fine by me. The coverage half of F7 is still open: the test exercises the deploy-returns contract but not the redeploy-vs-`taskDone` race on `cancellationFutures`. That depends on F1/F5 below.
3. **F4 / F6 / F8** — To make sure I read you correctly: "not touched by this diff" means the incremental push didn't change that code, not that it's outside this PR? Those findings were raised against code introduced or changed by this PR (the teardown under the service-wide monitor, the stale-cleanup branch, `finishExecutionContext`), so unless that code has been removed from the PR they remain open. If you think any of them is pre-existing behaviour, please point me at what shows that and I'll re-check.
Still open before I can approve:
- **F1 / F3**: does `deployLocalTask()` now install the context and cancellation future under the same monitor that `finishOwnedResources()` uses, or has the Javadoc been adjusted to match the actual locking? Neither shows up in your grep of the incremental diff, so I assume both are unchanged.
- **F5**: is the plain `put` in `deployLocalTask` still there? If so, I'd like either an explicit replace with a log line, or a short note on why overwrite is safe now that the worker no longer reads from the map.
- **F7 (coverage)**: if F1/F5 need a code change, a test exercising redeploy-while-`taskDone` on the same location would close this out; if they're argued safe, a comment in the test pointing to that argument is enough.
Happy to take another look once those are addressed.
<!-- streview-comment:1097 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5617534909
@SEZ9 — pushed f1e02f942 with the F3 rewording and the F8 comment.
On F3: I think the original text was accurate, but clearly not clear enough.
deployTask(TaskGroupImmutableInformation) opens synchronized (this) at line 528
and the deployLocalTask(...) call at line 550 sits inside that block, so the
redeploy guard and the deploy really do run under the same monitor as this
teardown — the claim was about deployTask, not deployLocalTask. I've reworded
it to name the enclosing block explicitly, say plainly that a caller reaching
deployLocalTask directly isn't covered, and point at #12164 for that path.
On F8: added a comment at finishExecutionContext explaining the stale branch
deliberately keeps its context out of finishedExecutionContexts, since the
location is reused across generations.
On F7: yes, the tests still use reflection — ReflectionUtils.getField for
executionContexts, cancellationFutures, taskAsyncFunctionFuture and
timerFlushFutures. I didn't add a non-reflective seam; happy to if you'd prefer
one, though it would mean widening the production API for tests, so I'd rather
you decide that than assume.
F1/F5 note added to the PR description, alongside the #12224 reference.
Locally both classes are green on this head: 20 tests, 0 failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5612012262
@abdessalems Confirming your recap matches my side too — F8's stale-branch semantics and the #12224 `CooperativeTaskWorker` deferral are as I described in my last comment, and I have no further source-side blocker on `e66911c0c`. Nothing more from me; over to @SEZ9 for the re-review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5597500536
Filed the CooperativeTaskWorker follow-up as #12224 — same generation-scoping
gap as BlockingWorker but on the thread-share path, only reachable under
task_execution_thread_share_mode: ALL/PART. I'll pick it up next, alongside
#12164.
Also re-ran just the two failed jobs (all-connectors-it-7 and
doris-connector-it) rather than the whole workflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5642644986
Follow-up on the `connector-rocketmq-e2e` failure I flagged as "looks unrelated" in my last comment: I pulled the actual job logs rather than just the job name. Both `rocketmq-connector-it (8)` and `(11)` fail across nearly every `RocketMqIT` test case with the same root error:
```
org.apache.rocketmq.client.exception.MQClientException:
The broker[broker-a] not exist
```
This is a topic-route-propagation race in the RocketMQ test container/client startup, not anything touching this PR's `TaskExecutionService` diff.
I checked whether `dev` already has a fix, and it does: commit `a9cda80` (landed 2026-09-10, after this branch's fork point) adds a `waitForTopicRoute("test_topic_source")` call in `RocketMqIT.startUp()` specifically to close this race, alongside a related `RocketMqConsumerThread`/`RocketMqSourceReader` `close()` fix. This branch (currently 97 commits behind `dev`) doesn't have that commit yet.
So the lowest-cost next step is to sync with the latest `dev` and rerun CI — that should clear this specific failure. If `connector-rocketmq-e2e` still fails after the sync, it would be worth a closer look, but I'd expect it to go green.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5642957326
Thanks for the update on `f1e02f942`. I haven't yet re-verified each of the earlier findings against this head, so rather than guess, could you note for each one whether it was addressed in `f1e02f942` or is intentionally deferred?
- **F1 / F3** – whether the context and cancellation-future installation on the deploy side now happens under the same monitor that `finishOwnedResources()` uses (closing the redeploy-vs-taskDone race), or whether the Javadoc was adjusted to describe what is actually guaranteed.
- **F2** – whether `BlockingWorker` now resolves its context from the tracker's `ownedContext` instead of the shared `executionContexts` map, so a reused `TaskGroupLocation` cannot hand it a newer generation's context/class loader.
- **F4 / F5** – whether the stale-cleanup path now cancels the old generation's async functions and timer-flush tasks, and whether installing the new context still uses a plain `put` that could overwrite a still-live older entry. Handling both in one place (detect the live older generation, tear its resources down, then install) would resolve them together.
- **F6** – if the teardown runs under the service-wide monitor, please consider moving the class-loader release and cancellation work outside the locked section and keeping only the map mutations under the lock.
- **F8** – `remove(key, value)` alone is sufficient, so the preceding `get` can go. Please also say whether the stale path should move `ownedContext` into `finishedExecutionContexts`, or why it intentionally does not.
**F7** – deferring the redeploy-vs-taskDone race test is fine with me as long as the PR description records that explicitly; the reflection-based access into `TaskExecutionService` internals is acceptable for now.
Once I have that per-finding status, I'll do a final pass on the head.
<!-- streview-comment:982 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5738518761
@abdessalems — thanks for pushing the F7 breadcrumb in 06c2ef03e. A class-level Javadoc on the race test stating that the redeploy-vs-taskDone half lives in `deployLocalTask()` and the tracker teardown, and that neither is touched in this PR, is the shape we discussed.
I haven't re-read 06c2ef03e yet, so I'm keeping F7 open until I've verified the Javadoc addition in the diff itself and re-confirmed that the rest of the change is still the 3-hunk diff I reviewed on 2dc21468105 (import removal plus the two `BlockingWorker` hunks). If that checks out, F7 is resolved from my side.
F2 remains open and will likewise be confirmed against the diff. Nothing further is needed from you on either item right now.
<!-- streview-comment:1150 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5728629462
@abdessalems — thanks for laying out the grep evidence so precisely, that resolves the ambiguity for @SEZ9 (see my reply to them above, which I cross-checked independently against the current head rather than just taking it on faith).
No new commit has landed on this PR since my last pass on `2dc21468105`, so my source-level review of the current 3-hunk diff (import removal + the two `BlockingWorker` hunks) stands unchanged for now.
On your F7 offer: adding a short comment in the race test pointing at where the `deployLocalTask()`/`taskDone()` redeploy race actually lives (i.e., in `dev`'s existing code, out of this PR's scope, tracked separately against #12238) sounds like the right, low-risk way to close that thread out without pulling unrelated `dev` behavior into this diff. Please go ahead and push it whenever convenient — once it's in, I'll take another look at that specific addition (and re-confirm nothing else moved) before circling back with @SEZ9. No other action needed from you on my end right now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5469381742
@SEZ9 — your Issues 1 and 6 on #11757 look like they may apply here too. The
stale branch of finishOwnedResources() calling recycleClassLoader() while the
old generation's async and timer-flush futures are still running came from my
commit originally, and #11727 carries the same call.
Worth me addressing it on this PR, or better handled once in #11757 given how
much the two overlap? Happy either way, just don't want it fixed twice or
missed entirely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5789685586
@davidzollo — following up on this one.
CI on a80834ebf (the head after your dev merge): all four unit-test legs green,
72 jobs passing. engine-v2-it is red on both legs, but that's the
SplitClusterFaultToleranceIT race @DanielLeens traced to dev's own code, tracked
in #12311 and still unmerged — your dev sync not clearing it is itself evidence
it isn't from this PR. The rest are the usual connector ITs.
The only thing still blocking merge is procedural: the PR carries
CHANGES_REQUESTED from @DanielLeens (09-07) and @SEZ9 (08-28). Both were resolved
through later COMMENTED updates rather than a formal state change, and
@DanielLeens has said he can't update his own because his account is
comment-only. Would you be able to dismiss those so the gate clears?
Happy to keep waiting if @SEZ9 still wants to finish verifying F2 and F7 first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5658073767
@abdessalems thanks for the rebase and for narrowing this back to the original scope.
I can't mark the earlier findings resolved from the thread alone, so I'll hold on that until I've gone through the updated diff:
- F1, F2, F4, F5, F6, F8: you note these all targeted the ownership model from the dropped commits (588314d9c, 1fed68145, 912e8bd4c). Once I've confirmed in the diff that none of that code remains here, I'm fine with those moving to the follow-up discussion you mentioned.
- F3: I'll verify the Javadoc with the "same monitor as deployment" claim is no longer present.
- F7: I'll check that the test no longer uses reflection into internals. On the coverage side, your framing sounds right to me — if the stale-taskDone trigger can't be demonstrated, the test and description shouldn't claim to cover it. Describing the latch release and the close()/initAttempted guard as the always-reachable fixes and the context resolution as hardening is a fair way to put it.
Two small asks:
1. Push the PR description and test Javadoc update you mentioned so they match that framing.
2. Ping me once that's up and I'll do a full pass on the remaining files.
<!-- streview-comment:1023 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5564126990
Thanks @DanielLeens for the CI triage on 23bbcb526, b555e0a9a and 18e821c3d — noted that the failures listed there are outside this PR's diff and that the retriggers (b555e0a9, 18e821c3) were empty commits with no code change.
With no code change in those retriggers, the earlier review findings on `TaskExecutionService` and `TaskDeployStaleContextRaceTest` remain open. To keep the review moving, here is what I still need:
1. **F1 / F3 / F5 (redeploy vs. taskDone race, monitor claim, plain `put`)** — please have `deployLocalTask()` install the context and cancellation future under the same monitor that `finishOwnedResources()` uses, and replace the plain `put` with a guarded install that refuses to silently overwrite a still-live older generation. Update the `finishOwnedResources()` Javadoc so it matches the actual locking once that is done.
2. **F2 (BlockingWorker context resolution)** — resolve the context from the tracker's `ownedContext` rather than the shared `executionContexts` map, so a reused `TaskGroupLocation` cannot pick up a newer generation's context/class loader.
3. **F4 / F8 (stale-cleanup branch)** — in the stale path, cancel the old generation's async functions and timer-flush tasks in addition to recycling class loaders, move `ownedContext` into `finishedExecutionContexts`, and drop the redundant `get` before `remove(key, value)` in `finishExecutionContext()`.
4. **F6 (heavy teardown under the service-wide monitor)** — please keep only the map/future bookkeeping under the `TaskExecutionService.this` monitor and run class-loader release and cancellations outside of it.
5. **F7 (test)** — extend `TaskDeployStaleContextRaceTest` to exercise the redeploy-vs-taskDone race on `cancellationFutures`, and if possible reduce the reflection into `TaskExecutionService` internals.
Once those changes are pushed, a short note here summarizing how each finding was addressed would be great, and I'll re-review promptly.
<!-- streview-comment:851 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5558122772
CI status (maintainer triage, not a review of the change): the Build on b555e0a9a finished with two failures, neither in this PR's diff: `DorisSchemaChangeIT` (MODIFY COLUMN cancelled the open stream load and the pipeline restart failed, #12131) and the `KuduIT` Flink JobManager hang cancelled at 90 minutes (#12132). Retriggered CI with an empty commit (18e821c3); no code change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5245916977
Follow-up filed for the underlying race: #11755 - a stale `taskDone()` removing a newer generation's execution context. This PR guards every read of that map; #11755 tracks the eviction itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5236051674
@DanielLeens I'll take care of the test comment and Javadoc updates.
For Issue 1, I think it makes sense to include the null check in
`recycleClassLoader()` in this PR, since this failure path is now reachable
because of the fix. But I'm happy to split it out if you prefer to keep this PR
as small as possible.
I'll also open a follow-up for the underlying `taskDone()` /
`executionContexts` race and link it here and to #11679.
Once these are done, I'll take the PR out of draft.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5241810696
Taking this out of draft - all review items are addressed.
CI note: two red jobs on the current head, neither from this change.
- `Dead links` fails on a 404 from a MongoDB docs URL in the repo's markdown.
This PR touches no markdown.
- `unit-test (11, windows-latest)` fails on
`MultiTableSinkWriterSchemaChangeBroadcastTest.schemaChangeKeepsIOExceptionContractWhenWorkerAlreadyFailed`
in `seatunnel-api` (383 tests, 1 error). The same test failed the same way on
the previous head before this test existed, and the reactor stops there, so
`seatunnel-engine-server` is never reached. The other three `unit-test` legs
were cancelled by matrix fail-fast rather than failing on their own.
Happy to rebase or re-run if that helps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5417359245
@DanielLeens @davidzollo — thanks both, though I think you've pointed me in slightly different directions, and I'd rather ask than guess again.
Daniel, you suggested landing this PR with just the getClassLoaders() guard and leaving the eviction and ownership work entirely to #11757. David, you'd prefer the ownership fix here — the tracker holding the TaskGroupContext it owns, with an identity-checked remove(location, ownedContext) — and #11757 rebasing on top, and you said you wouldn't merge this without it since this PR is what makes taskDone() reliably reachable on the failure path.
That last point makes sense to me, so unless Daniel objects I'll go with David's direction. I'd keep it tight: just the tracker-owned context and the identity-checked remove, plus a cross-generation regression test. I'd leave the broader teardown work — cancellationFutures, async functions, timer flush, the synchronized block — to #11757, since that's clearly the more complete design there.
@waterWang — sorry for the overlap, it wasn't intentional. If it lands this way, #11757 would need a rebase on top of this. Happy to coordinate on where the line sits so we don't write the same thing twice, and if you'd rather keep the identity check on your side I'm fine with that too — just say and I'll leave it out here.
Holding off on pushing until this is settled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5422721900
Pushed. I cherry-picked the three ownership commits from #11757 rather than rewriting them, so @waterWang's 614b3c097 and your 6f8382a88 / a8407319b keep their original authorship — thanks both, that saved doing it twice.
On top of that I added one commit with the two things that were still mine: the getClassLoaders() null guard in BlockingWorker.run(), and the leak you diagnosed — finishOwnedResources() was returning from the stale branch without releasing the loaders this tracker installed, so I call recycleClassLoader(taskGroupLocation, ownedContext) before that return.
Locally TaskDeployStaleContextRaceTest passes (2/2) and testStaleTaskDoneDoesNotCleanupNewerGenerationResources passes. Spotless applied. The rest of TaskExecutionServiceTest I couldn't run here — surefire's process checker fails on Windows and kills its own fork on a ping timeout — so CI is the real check on that class.
@waterWang — #11757 will want a rebase on top of this now. Sorry for the extra step, and happy to help if it's awkward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5419385522
@abdessalems @davidzollo - sorry for the mixed signals, and thanks for holding off and asking rather than guessing again. I went back through both diffs (not from memory) before answering.
I'm changing my recommendation: **David is right, and I'd go further than the "tight" split proposed above.**
Why the reversal: the reason this matters more than a normal scope call is that #11727 itself changes when `taskDone()` runs. On `dev` today, the classloader lookup happens *before* the `try/finally`, so a null context throws before `taskGroupExecutionTracker.taskDone(t)` is ever reached - the stale-generation cleanup path in `taskDone()` is unreachable from this failure. #11727 moves that lookup inside the `try`, so the same null-context case now falls through to `finally` and calls `taskDone()` for the first time. That's a real, new trigger for the #11755 race, introduced by this PR's own fix - not a hypothetical shared by both PRs equally. David's "the cleanup path needs to be generation-safe before merge" is correct on the code, not just a preference.
One correction to the "tight" version floated above (owned-context + identity-checked remove only, leaving the rest to #11757): that's not sufficient on its own. In the current `taskDone()` body, `cancellationFutures.remove(taskGroupLocation)`, `cancelAsyncFunction(taskGroupLocation)`, and `cancelTimerFlushForTaskGroup(taskGroupLocation)` all run unconditionally, straight after the context/classloader recycle step. If only the context swap is identity-gated, a stale tracker can still drop the newer generation's cancellation future and cancel its async functions and timer-flush tasks - same corruption class, different door. That's exactly why #11757 wraps all five steps (classloader recycle, context handoff, cancellationFutures removal, async-function cancel, timer-flush cancel) inside one identity-checked `synchronized` block (`finishOwnedResources()` / `finishExecutionContext()`), not just the context piece.
Concrete recommendation: don't reimplement a narrower subset in #11727. Port the `ownedContext` field and the `finishOwnedResources()`/`finishExecutionContext()` pair from #11757 wholesale - it's already written, already covers the full bundle, and already has its own regression test (`testStaleTaskDoneDoesNotCleanupNewerGenerationResources`). @waterWang, would you be up for either pushing that commit directly onto abdessalems' branch or letting him cherry-pick it with credit, so it isn't written twice? Once that's in #11727, #11757 rebases on top and should shrink to whatever's left beyond the ownership guard, if anything.
Keep the `getClassLoaders()` null-guard and the finally-block latch release in #11727 as-is - those are still correct and useful as a defensive backstop even with the ownership check in place. This should be the last open question before this is mergeable; thanks again for the patience through all the rounds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5553290460
Retrigger pushed: `5a442e4` (empty commit, by davidzollo). The run on `a30a0bd7b` (fork run 33963286570) finished 75 success / 14 skipped with exactly the three failures already triaged above — `DorisErrorIT` stop/DNS race (source fix #12114), `DorisSchemaChangeIT` post-DDL stream-load lag, and the `PostgresCDCIT` committed-offset restore skipping a row (known #11847, fix #11864 still open) — plus a fail-fast `kudu-connector-it` cancel. None touch this PR's diff. Expect the Postgres one to recur intermittently until #11864 lands; the Doris race clears once #12114 lands and is synced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5473887244
Thanks @DanielLeens — that's a really useful breakdown on the cacheMode
default, and agreed on leaving it as a non-blocking follow-up rather than
fixing it in both PRs.
@davidzollo @zhangshenghang — this looks ready from the review side: Daniel's
conclusion is "ready to merge", the unit-test legs are green on the current
head, and the only red jobs are connector ITs unrelated to the two files this
touches. Could one of you take the approve/merge step when you have a moment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5381239640
Thanks @abdessalems, and thanks @davidzollo for the +1.
To recap: my from-scratch re-review of the current head (`72268def`) found no blockers — the core fix (lookup moved inside `try`, explicit `IllegalStateException` instead of the implicit NPE, `startLatchReleased`-guarded latch release in `finally`) and the null guards on the racy `executionContexts` reads in `taskDone()` and `recycleClassLoader()` are all in the shape I reviewed at `45a6d3bf7`, plus the `TaskDeployStaleContextRaceTest` covering the stale-generation race.
Since we agree this stays as the main PR, I'll close the duplicate as superseded.
Remaining before merge: let CI finish on the fresh base you pushed after merging latest dev. If anything fails, ping here or the #contributors Slack channel davidzollo linked; otherwise no further code changes are needed from my side.
Once CI is green this is good to merge. Thanks for the persistence, @abdessalems — turning a silent client hang into a loud, bounded failure is a real quality-of-life win.
<!-- streview-comment:456 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5825623520
Thanks @abdessalems. To be clear, this doesn't change what I said above: the source diff on `a80834ebf` is unchanged from `06c2ef03e`, the approval stands, and I don't think the merge needs to wait on the unrelated failures.
One housekeeping question, not a gate: could you confirm whether the points from my earlier pass on `TaskExecutionService` (the deploy/teardown monitor, the `BlockingWorker` context source, and the stale-cleanup path) were addressed at `06c2ef03e`? Since the current head doesn't touch those files, I just want to make sure I'm not carrying a stale view of them. If they were resolved, nothing further is needed from my side; if any are still open, a pointer to where they stand would be enough.
<!-- streview-comment:1294 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5390960726
Thanks @DanielLeens for the thorough follow-up, and for verifying that the merge commit `51c6a46a` on top of `72268def` is a no-op on `TaskExecutionService.java` and `TaskDeployStaleContextRaceTest.java` — that saves everyone a redundant re-review pass.
I agree both points you've carried forward are in scope for this PR and should be fixed here rather than deferred, since they sit in the same method and the same stale-generation race family this fix targets:
1. **Cross-generation eviction race** — the suggested direction sounds right to me: `taskDone()` should capture `TaskGroupContext finishedContext = executionContexts.remove(taskGroupLocation);` (or better, an ownership-aware `remove(key, value)` with an identity check against the context pinned at deploy time) and recycle only that specific instance, instead of `recycleClassLoader()` re-resolving by key with its own `executionContexts.get(taskGroupLocation)`. Key-only cleanup is unsafe precisely because `TaskGroupLocation` is reused across restore generations, which is the premise of this fix.
2. **NPE / silent-null class loader path** — `taskGroupContext.getClassLoaders().get(t.getTaskID())` in `BlockingWorker.run()` needs a guard after the existing null-context check, and a missing per-task entry should fail fast rather than silently installing a null context class loader.
Concrete asks before merge:
- Push a fix commit addressing both points (ownership-aware cleanup in `taskDone()`/`recycleClassLoader()`, plus the guard/fail-fast in `BlockingWorker.run()`).
- If feasible, extend `TaskDeployStaleContextRaceTest` (or add a companion test) to cover the cross-generation scenario where a newer generation installs its context under the same key before the older generation's `taskDone()` runs, so the eviction race stays regression-tested.
@DanielLeens once the fix commit lands, a fresh pass from you on just those two locations would be much appreciated. And as mentioned earlier, feel free to ping the #contributors channel on the SeaTunnel Slack if CI gives you any trouble.
<!-- streview-comment:518 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5728628355
@SEZ9 — I independently re-checked the current head (`2dc21468105`) before replying, rather than just taking @abdessalems's word for it:
- The production diff in `TaskExecutionService.java` really is exactly 3 hunks: the import removal (`@@ -73,7 +73,6 @@`) and the two hunks inside `BlockingWorker`'s constructor/`run()` (`@@ -1277...@@`, `@@ -1308...@@`). Neither touches `deployLocalTask()`'s body — the only occurrences of `deployLocalTask` anywhere in the full diff are new call sites in `TaskDeployStaleContextRaceTest.java` and `TaskExecutionServiceTest.java`, which exercise the existing method rather than change it.
- `finishOwnedResources`, `finishExecutionContext`, and `cancelOwnedAsyncFunctionsInPlace` all come back with 0 hits against the current diff, matching your grep. That ownership model is genuinely gone — it lived in the three commits cherry-picked from #11757 (closed unmerged), and #12238 replaced it with a different, identity-checked `compute()` design.
So on the ambiguity you raised: @abdessalems's "not touched by this diff" does mean "no longer part of this PR at all," not "unchanged since the last push." F4/F6/F8 targeted code that simply isn't in this PR anymore — if you still think they apply, that would need to be raised against #12238 itself, not here.
F1/F3/F5 are a genuinely separate question: they're about `dev`'s pre-existing `deployLocalTask()` locking/`put` semantics, which this PR doesn't touch in either direction. I'd treat those as valid `dev`-level concerns to track against #12238 rather than blockers on this specific 3-hunk PR, but I understand wanting more than an argument to rely on — a short comment in the test pointing at exactly where that race lives (as offered for F7 coverage) seems like a reasonable, low-cost breadcrumb either way.
My source-level review of this PR's actual diff stands as I left it. Happy to take another pass once you've confirmed F2 against the diff yourself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5739948024
> @DanielLeens — pushed the F7 breadcrumb in [06c2ef0](https://github.com/apache/seatunnel/commit/06c2ef03e73199d7ea35932a47385b03969b2b3f). The race test's class Javadoc now states explicitly that the redeploy-vs-taskDone half lives in deployLocalTask() and the tracker teardown, that neither is touched here, and that it's #12238's design and tracked against it — so nothing in this test reads as covering it.
>
> Thanks for re-checking the diff independently rather than taking my grep on faith.
>
> @SEZ9 — that closes the F7 coverage thread from my side. F2 is the one item left for you to confirm against the diff.
+1
LGTM if CI passes
This PR is about 40 commits behind the latest code. I merge
the upstream code to make it better to get through CI check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5674932827
Thanks @DanielLeens for tracking those down. One precision point for @SEZ9's
scope, since both answers describe code that's on the current head but came in
via #12238, not this PR's diff:
1. testStaleTaskDoneCleansOnlyOwnedGenerationResources is already on dev from
#12238 — this PR doesn't add it. The only test method #11727 adds to
TaskExecutionServiceTest is testMissingTaskClassLoaderFailsFast, plus the
helper fix. So the redeploy-vs-taskDone coverage on cancellationFutures is
provided by #12238, which is consistent with where F1/F5 were deferred.
2. finishOwnedResources() doesn't appear in this PR's diff at all, so there's no
monitor claim of mine left to check — that Javadoc is #12238's.
What this PR actually changes: BlockingWorker.run() in TaskExecutionService.java,
TaskDeployStaleContextRaceTest (the deploy-returns contract), and the one test
above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-6383278055
Thanks @SEZ9 for the careful follow-up. I fully agree on the CI part: the `engine-v2-it` failures (`BackpressureSlowSinkIT` and `CheckpointCoordinatorFailoverIT`) sit close to the task-lifecycle code, so a same-scenario baseline run on `dev` is still needed before anyone calls them unrelated. That ask stands for @abdessalems.
On the source items, I re-read the current head `a80834ebfb185ee0d54243e7a3ae99a5bf481817` again, and several of the points still listed as open describe code that is not in this version. To keep the thread from drifting, here is what this head actually does:
- **F1 / F3 / F5 (deploy vs. taskDone):** there is no `finishOwnedResources()` method and no Javadoc in this head that claims a shared `TaskExecutionService.this` monitor; `git grep` finds neither `finishOwnedResources` nor `synchronized (TaskExecutionService.this` in the file. The deploy side is documented as lock-free on purpose ([lines 705-722](https://github.com/apache/seatunnel/blob/a80834ebfb185ee0d54243e7a3ae99a5bf481817/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/TaskExecutionService.java#L705-L722)). It registers the execution-scoped resources first and publishes `location -> context` last. `cancellationFutures` is keyed by `TaskGroupContext` ([line 217](https://github.com/apache/seatunnel/blob/a80834ebfb185ee0d54243e7a3ae99a5bf481817/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/TaskExecutionService.java#L217)), not by location, so a new deployment's `put` cannot overwrite an older
generation's cancellation future. The only location-keyed `put` is the `executionContexts` publish, where a newer generation is meant to supersede the older one. The older tracker then cannot remove it, because `finishExecution()` only clears the mapping when `context.equals(activeContext)` inside `executionContexts.compute(...)` ([lines 1689-1701](https://github.com/apache/seatunnel/blob/a80834ebfb185ee0d54243e7a3ae99a5bf481817/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/TaskExecutionService.java#L1689-L1701)).
- **F2 (BlockingWorker context):** `BlockingWorker.run()` reads `taskGroupExecutionTracker.context.getClassLoaders()` ([lines 1285-1294](https://github.com/apache/seatunnel/blob/a80834ebfb185ee0d54243e7a3ae99a5bf481817/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/TaskExecutionService.java#L1285-L1294)), which is the context the tracker pinned at deploy time. It does not look anything up in the shared `executionContexts` map.
- **F4 (stale-branch cleanup):** the stale branch does not return early. After the conditional `compute`, `finishExecution()` always removes this context's cancellation future, recycles its class loaders, and cancels its async-function and timer-flush futures ([lines 1703-1728](https://github.com/apache/seatunnel/blob/a80834ebfb185ee0d54243e7a3ae99a5bf481817/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/TaskExecutionService.java#L1703-L1728)). The superseded case only logs a warning.
- **F8 (`finishedExecutionContexts`):** only the active generation is moved into `finishedExecutionContexts`. That map is keyed by `TaskGroupLocation`, so putting a superseded generation there would record a stale context under a location the newer generation owns. If you think the superseded generation should be recorded somewhere, I am open to discussing it, but this is a design choice and not a missing cleanup step. There is also no `finishExecutionContext()` method with a redundant `get` in this head.
- **F6 (teardown under the service monitor):** none of the teardown runs inside a service-wide `synchronized` block. The only atomic section is the per-location `compute`, and class-loader recycling plus future cancellation run after it.
- **F7 (test coverage):** this one I agree with as a reasonable non-blocking improvement. `TaskDeployStaleContextRaceTest` states that it covers only the deploy-return contract ([lines 65-85](https://github.com/apache/seatunnel/blob/a80834ebfb185ee0d54243e7a3ae99a5bf481817/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/TaskDeployStaleContextRaceTest.java#L65-L85)). A direct redeploy-vs-`taskDone()` test using package-private hooks instead of reflection would strengthen it.
If any of F1-F6/F8 refers to a specific line on this head that I am reading differently, please point to the exact line and I will gladly re-check it. Otherwise, I think the remaining open items for @abdessalems are the baseline comparison for the two engine IT failures and, optionally, the F7 test. This is not a new approval or a claim that the PR is ready to merge while `Build` is still red.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5805306026
Thanks for the follow-ups, @SEZ9 and @abdessalems.
I independently diffed the three files this PR actually touches (`TaskExecutionService.java`, `TaskDeployStaleContextRaceTest.java`, `TaskExecutionServiceTest.java`) between the head I last fully reviewed (`06c2ef03e`) and the current head (`a80834ebf`): the file-scoped diff is empty, so the dev-sync merge is confirmed byte-identical for this PR's own content. My "Ready to merge" technical conclusion from that review stands unchanged on `a80834ebf`.
@SEZ9 — re: F1/F3/F5/F6/F8: since the content is unchanged, my answer from the `06c2ef03e` review still applies as-is: `git grep` for `finishOwnedResources`, `finishExecutionContext`, `cancelOwnedAsyncFunctionsInPlace`, `ownedContext` and `synchronized (TaskExecutionService.this)` still returns zero hits on this head, and `deployLocalTask()`'s body is still untouched by this PR. Those five items remain scoped to `dev`/#12238, not this 3-hunk fix. F2 and F7 are confirmed as before.
@abdessalems — I checked `reviewDecision` on the PR directly rather than relying on the sidebar: it currently reports `APPROVED`, so GitHub's aggregate decision is not actually blocked by any reviewer's state right now (the `CHANGES_REQUESTED` entries you both see are historical display, not the active gate). The `mergeStateStatus` is `BLOCKED`, and that's the required `Build` check, which is still red only from the known `SplitClusterFaultToleranceIT` flake tracked in #12311 (still open/unmerged upstream, unrelated to this diff, as I confirmed in my last comment). So there's nothing left for me to dismiss or update — the remaining gate is CI, not review state.
No new technical findings this round; my last full review (on `06c2ef03e`) remains the complete and current conclusion for this PR's content.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5245888290
@DanielLeens - closing out your Issue 1. You were right that no CI leg had
actually reached `seatunnel-engine-server`: I checked all four `unit-test` legs
on this head and the module never ran on any of them (the Windows JDK 11 leg
stops in `seatunnel-api`, and fail-fast cancelled the other three).
Rather than re-run the full 82-job matrix, I ran the module on its own against
this exact head, Ubuntu / JDK 8:
```
maven.compiler.target=1.8
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0 - TaskDeployStaleContextRaceTest
BUILD SUCCESS
```
Run: https://github.com/abdessalems/seatunnel/actions/runs/31430929790
Both Java files there are byte-identical to e9b25d8 (`TaskExecutionService.java`
sha256 `c0ae4127...`, `TaskDeployStaleContextRaceTest.java` sha256 `e8661d5c...`);
the only difference on that branch is the workflow file used to launch the run.
On the rest of that build: the four other failures are all outside this diff's two
files - `MultiTableSinkWriterSchemaChangeBroadcastTest` in `seatunnel-api`,
`JdbcPrestoIT`, `MysqlCDCWithFlinkSchemaChangeIT`, and a RocketMQ container
failure with no test-level error. The `seatunnel-api` one fails identically on the
previous head, before this test existed.
Filing the follow-up for the underlying race next and will link it here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5601140161
@abdessalems thanks for filing #12224 for the `CooperativeTaskWorker` gap and linking it here alongside #12164 — that was the one remaining ask on my side, so I have no further source-side blocker on this PR. Nice work re-running just the two unrelated failed jobs instead of the whole workflow too.
@SEZ9 — re: the truncation, I re-pulled my own comment's stored body directly via the API (not the web renderer) and it is not cut off there either: after `executionContexts.remove(taskGroupLocation, ownedContext)` — it continues "a compare-and-remove keyed on the exact context object this tracker owns. That call is safe against *any* current map value: if the map holds something other than `ownedContext` (a leaked entry, a newer generation's context, or nothing at all), the remove simply returns `false` and `finishOwnedResources` takes the "stale" branch (`:1571-1576`), preserving whatever is there." So — same as the rendering artifact you hit on my end earlier in the thread — this looks like a display-side truncation on your end, not a gap in what's actually stored.
To answer your question directly, since it matters more than the transport glitch: **the `remove(key, value)` path tolerates a leaked entry being present.** It's a compare-and-remove, not an unconditional remove, so an unexpected value (leaked entry, a different generation's context, or absence) all fall through to the same "stale, do nothing to this entry" branch rather than corrupting state or throwing. The ownership logic's correctness does not depend on the leaked entry being absent — it was written to tolerate exactly that case.
What the leak does break is upstream, not in this PR's own logic: a leaked `executionContexts` entry from a submit-time failure (#12164) means no `TaskGroupExecutionTracker` for that generation ever runs to call `finishOwnedResources()`, and `deployTask`'s outer guard treats any live entry as "already active," so no redeploy can even create the next generation to own that slot. This PR's ownership path is correct in the presence of the leak, just starved by it — which is why the leaked-entry point stays folded into the Blocker 1 tracking split (independent bugs that happen to compound in one failure window) rather than needing its own rollback in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5649652966
@abdessalems Thanks for flagging this explicitly rather than letting the two land in conflict — that's the right call to ask for.
I checked both PRs' file lists directly rather than going from memory: #12238 touches `TaskExecutionService.java` + `TaskGroupContext.java` (its re-keying model) plus its own new test, while this PR touches `TaskExecutionService.java` plus `TaskExecutionServiceTest.java` — the same test file #12238 also modifies. So the overlap isn't just conceptual (two ownership models for the same problem), it's a literal same-file, same-test-file collision if both land as-is.
My recommendation: let #12238 land first. It's already gone through a full review round from me with no outstanding code-side blocker and green CI, and re-keying by `TaskGroupContext` is the more self-contained fix — it doesn't need to introduce a new tagging concept (`OwnedFuture`) into `TaskExecutionService`, it scopes the fix to the actual execution-context boundary. Once it's in, I'd ask you to rebase this PR back to its original, narrower scope — the unconditional `startedLatch` release for #11679 — and drop the three cherry-picked `OwnedFuture` commits, since #12238 will already have solved the ownership-tagging problem they were addressing. That keeps this PR focused on the bug it originally set out to fix and avoids maintaining two parallel ownership models in the same methods.
To be clear, the final call on sequencing is a maintainer decision (looping in @davidzollo and @SEZ9 as you did) — this is my technical read based on the actual diffs, not an official ruling. But from where I sit, "strip this back to the latch fix and let #12238 carry the ownership-model fix" is the cleaner outcome for the codebase. Appreciate you keeping this constructive instead of just letting the merge conflict happen — that made this a much easier question to answer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5653016793
Thanks @DanielLeens — done, and you were right that it removes code rather than
adds it.
Issue 1: the class loader now comes from taskGroupExecutionTracker.context
instead of a fresh executionContexts lookup. Since that's pinned as a @NonNull
field at construction, the null-context branch is gone entirely — net -7 lines.
Issue 2: agreed, and I'd rather state it accurately than overclaim. Post-#12238
I can't point at a currently-reachable path that produces the original
"stale taskDone() deletes the newer context" trigger either, for the reasons you
gave — deployTask()'s synchronized + containsKey guard and the identity-checked
compute() in finishExecution(). So this is better described as: the latch release
and the close()/initAttempted fix are independent, always-reachable corrections,
and the context resolution is hardening. I'll update the PR description and the
test Javadoc to match rather than describe a trigger I can't demonstrate.
Locally green: 21 tests, 0 failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5441639543
CI is back, @DanielLeens — all four unit-test legs completed and passed this time: 50/46/60/61 min, versus hitting the 90-minute cap on the last two heads.
The root cause was the getClassLoaders() guard from the previous round, although the guard itself is correct. The test helper only registered a class loader for the first task in a group, so tests deploying two tasks left the second without one. The helper now mirrors deployTask() and registers one per task.
Locally and in CI, the relevant tests are now green. I also ruled out the other two candidates: removing the synchronized block made no difference, while removing the recycleClassLoader call caused a hang.
The only remaining red job is all-connectors-it-1 (8, ubuntu-latest), which was already failing on 51c6a46a before these changes and is unrelated to seatunnel-engine-server. kudu-connector-it was cancelled behind it.
@waterWang — also worth flagging for #11757: its unit-test job was skipped, so this issue wouldn't have been caught there either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5467166436
Pushed, @DanielLeens — ITERATIONS is back to 30.
While I was in there I noticed the same merge had also dropped
DEPLOY_TIMEOUT_SECONDS from 30 to 10 in that file, so I restored both. That
one felt worth putting back: the timeout is the assertion the test rests on,
and at 10s a slow-but-healthy deploy on a busy runner would report a hang that
isn't there — flaky against exactly the bug the test guards.
Locally both classes are green at the restored values: 19 tests, race test
~20s at 30 iterations, so no timeout pressure either way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5775025119
@SEZ9 the F2 confirmation is in my review on commit 06c2ef03e: https://github.com/apache/seatunnel/pull/11727#pullrequestreview-5254567038, under the "Response to the open @SEZ9 items" section:
> F2: confirmed. The worker reads taskGroupExecutionTracker.context (:1294); that field is private final (:1561), assigned once in the constructor (:1568) from the same object that deployLocalTask() builds at :700-703. It is named context on this branch (not ownedContext). The shared-map lookup and its null-context branch are gone from BlockingWorker.
That review was on 06c2ef03e, before the dev merge that produced the current head (a80834ebf). As noted in the same review and confirmed by @abdessalems, none of the dev commits pulled in by that merge touch the three files changed in this PR, so the F2 finding still applies unchanged to the current head - nothing to re-verify there.
On the engine-v2-it failure: agreed, that is the dev-side SplitClusterFaultToleranceIT race tracked in #12311, not something this PR or the dev sync could fix, so I would not block on it either.
Thanks for tracking this down carefully across all these rounds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5787068092
@/tmp/reply_body_11727.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5855058822
@SEZ9 confirmed — yes, all three points (the deploy/teardown monitor, the `BlockingWorker` context source, and the stale-cleanup path) were addressed at `06c2ef03e`, and I already answered this exact question in my file-scoped diff comment above: the current head (`a80834ebf`) is byte-identical to `06c2ef03e` for the three files this PR actually touches, so nothing has moved since that review. You are not carrying a stale view — nothing further needed from your side either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5379413855
Following up here since this is the PR that will actually merge (per the discussion above — thanks @abdessalems for confirming #11910 should close as superseded, which I've now done).
Since #11910's head (`ffe51c08a551`) is byte-identical to this PR's current head (`72268def15f0`) on the two files that matter (`TaskExecutionService.java`, `TaskDeployStaleContextRaceTest.java` — re-verified via direct diff just now, zero output), a review posted on #11910 by @SEZ9 applies verbatim here and hadn't been forwarded yet. I independently re-read the current source at each location and confirm both of the following are real, unresolved issues that should be addressed before this merges:
**1. `recycleClassLoader()`/`taskDone()` ordering allows a stale generation to release a live generation's class loaders (Medium)**
- Location: `seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/TaskExecutionService.java:1421` (call site), `:1481` (re-`get()`)
- `taskDone()` calls `recycleClassLoader(taskGroupLocation)` *before* `executionContexts.remove(taskGroupLocation)`, and `recycleClassLoader()` independently re-resolves the context via its own `executionContexts.get(taskGroupLocation)` rather than operating on a reference the caller pins down. Because `TaskGroupLocation` is reused verbatim across restore generations — the exact premise this fix is built around — two different generations' trackers can each observe a non-null context here and each call `classLoaderService.releaseClassLoader(...)`. In the worst case, if a newer generation has already installed its own context under the same key by the time an older generation's stale `taskDone()` runs, the old generation's `recycleClassLoader()` nulls out and releases the *new*, still-in-use generation's class loaders — not just a double-release of the same acquisition.
- Suggested fix: have `taskDone()` capture `TaskGroupContext finishedContext = executionContexts.remove(taskGroupLocation);` first (the `ConcurrentHashMap` guarantees a single winner), then recycle only that specific non-null instance, instead of `recycleClassLoader()` re-resolving by key.
**2. `getClassLoaders()` can NPE between the null-context check and its use (Medium)**
- Location: `TaskExecutionService.java:1096` (dereference), `:1493` (concurrent null-out)
- `BlockingWorker.run()` null-checks `taskGroupContext` and then dereferences `taskGroupContext.getClassLoaders().get(t.getTaskID())` without a further guard. A concurrent stale `recycleClassLoader()` can call `context.setClassLoaders(null)` on that same instance between the check and the dereference — made more likely, not less, by issue 1 above — producing an undiagnosed NPE instead of this PR's own deliberate `IllegalStateException`. Separately, `.get(t.getTaskID())` returning `null` lets `setContextClassLoader(null)` succeed silently, so the task would run under the system class loader and fail much later with a confusing `ClassNotFoundException` instead of failing fast.
- Suggested fix: snapshot both the class-loader map and the per-task class loader into locals, and throw the same descriptive "context no longer registered" `IllegalStateException` when either is null.
Both are genuine correctness gaps in the exact class-loader lifecycle this fix is meant to make race-safe, not just polish. @SEZ9 also raised several Low-severity items (recycle-skip leaking the current generation's class-loader reference, the regression test's remover-thread fidelity/busy-spin, reflection vs. a package-private test accessor, and a `Collections.singletonList`/`Arrays.asList` nit) — see the full write-up on #11910 for those; I haven't re-derived them independently but have no counter-evidence against any of them.
None of this changes my assessment that the overall approach (moving context resolution inside `try`, guaranteeing the latch releases exactly once, guarding the now-more-reachable cleanup path) is the right fix for #11679 — it's a precise fix for the bug it targets. But issues 1 and 2 above are in the same method this PR touches and are part of the same race family, so I'd like to see them closed here before merge rather than filed as a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5425682018
@DanielLeens Thanks for the detailed review and for catching the Task.close() issue.
I pushed the fix in 8122bdbfa. I added the initAttempted guard so we only call close() after initialization was actually attempted, and widened the close-error handling so an unexpected exception doesn't escape the finally block.
I also added the regression test for the missing class-loader case. I verified it fails without the guard (30s timeout) and passes with it, and testStaleTaskDoneDoesNotCleanupNewerGenerationResources passes as well. Spotless is clean.
@waterWang the branch has moved again, so #11757 will need to rebase on this head.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5725650648
@SEZ9 — to answer directly: "not touched by this diff" means that code is no
longer in this PR at all, not that the last push left it unchanged.
Those findings were raised against the ownership model this PR carried while it
had the three commits cherry-picked from #11757 (588314d9c, 1fed68145,
912e8bd4c). That model was removed when I rebased onto dev and stripped the PR
back on @DanielLeens's and your suggestion. Checking the current head:
finishOwnedResources dev: 0 occurrences PR head: 0
finishExecutionContext dev: 0 PR head: 0
cancelOwnedAsyncFunctionsInPlace dev: 0 PR head: 0
They aren't on dev either — #11757 was closed unmerged, and #12238 solved the
same problem with a different design (finishExecution() with an identity-checked
compute()). So F4, F6 and F8 have no code left to apply to here.
Same for F1/F3/F5: this PR's diff contains zero occurrences of deployLocalTask.
The plain put you're asking about is dev's code, unchanged by this PR — if it
still needs addressing it belongs against #12238's design, not here.
The whole diff is three hunks: one import removal and two inside
BlockingWorker.run(). Happy to be shown otherwise if you spot something I've
missed.
On F7 coverage — since F1/F5 aren't in this PR, the redeploy-vs-taskDone race
lives entirely in dev's code. I can add a comment in the test pointing at that if
you'd like it recorded, as you suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5746921805
Thanks for the CI rundown on a80834ebf. Given the unit-test legs are green and the engine-v2-it failure is the dev-side SplitClusterFaultToleranceIT race tracked in #12311, I'm fine not blocking this PR on it — that isn't something the dev sync could have fixed.
On the two stale CHANGES_REQUESTED reviews: yes, I can dismiss them so the gate clears. Before I do, one thing I want to be sure about. From the thread, F7 was closed out with the Javadoc breadcrumb in 06c2ef0 (the race test now says explicitly that the redeploy-vs-taskDone half lives in deployLocalTask() and the tracker teardown and is tracked against #12238), but the last note I see on F2 was that it was still the one item left for the second reviewer to confirm against the diff. Could you link the comment where F2 (BlockingWorker resolving its context from the tracker's ownedContext rather than the shared executionContexts map) was confirmed? Once I have that, I'll dismiss both reviews and we can proceed with the merge.
<!-- streview-comment:1173 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "abdessalems (via GitHub)" <gi...@apache.org>.
abdessalems commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5252913785
@zhangshenghang @davidzollo — the PR is ready from my side whenever a committer has a chance to merge it. Thanks again!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-5881982649
Thanks for pushing for a real confirmation here, @SEZ9 - a three-item summary doesn't cover what you originally raised, fair call. I checked all four items directly against the current head (`a80834ebf`, byte-identical to `06c2ef03e` for the three files this PR touches):
1. **The plain `put` in `deployLocalTask` overwriting a still-live older generation's context/cancellation future** - this is explicitly out of scope for this PR, not resolved by it. `TaskDeployStaleContextRaceTest.java:80-85` says so directly: "the other half of the same family - a redeploy for this location racing an in-flight `taskDone()`, which can evict or tear down the newer deployment's cancellation future, async functions and timer flushes - lives in `deployLocalTask()` and the tracker teardown, neither of which this test nor the fix it covers touches... That path is the design of #12238 and is tracked against it." So: not addressed here, tracked separately under #12238. What this PR closes is the other half of the family - the deploy-side hang from #11679.
2. **Heavy teardown running under the service-wide `TaskExecutionService.this` monitor** - not the case at the current head. `finishExecution()` (`TaskExecutionService.java:1685-1729`) does the active-to-finished transition via `executionContexts.compute(...)` (line 1692), which only locks that map's internal bucket for that one key, not the service-wide monitor. `recycleClassLoader`, `cancelAsyncFunctionFutures` and `cancelTimerFlushFutures` (lines 1714-1728) all run afterward, outside any `synchronized` block. So class-loader release and async/timer-flush cancellation are not held under a service-wide lock.
3. **Race test covering the redeploy-vs-taskDone race on `cancellationFutures`, and the reflection** - same answer as (1): the test class javadoc says this race is explicitly out of scope for this PR and deferred to #12238; it isn't silently unaddressed, it's a documented boundary. On the reflection: `TaskDeployStaleContextRaceTest.java:280-289` gives the reason - `executionContexts` is private, and the only public ways to remove an entry (`deployTask`, `cancelTaskGroup`) tear down the whole task group rather than just the map entry, which would stop the deployment being raced instead of racing it. Adding a package-private accessor purely for this test would widen the production surface for test-only visibility, so the reflection is a deliberate, documented tradeoff rather than an oversight.
4. **The stale path not moving `ownedContext` into `finishedExecutionContexts`, and a redundant `get` before `remove(key, value)`** - first half is correct as read, and it is intentional: in `finishExecution()`'s `compute()` (lines 1692-1701), the stale branch (`!context.equals(activeContext)`) returns `activeContext` unchanged and never touches `finishedExecutionContexts` - only the branch that matches the currently active context moves itself into `finishedExecutionContexts` before returning `null`. A stale/superseded generation's context is deliberately dropped rather than archived, since it was never the result callers should observe for that location. Second half: I could not find a `get`-then-`remove(key, value)` pattern anywhere in `finishExecution()` or elsewhere in the file at this head - the only removal from `finishedExecutionContexts` is the unconditional single-key `remove` in `notifyCleanTaskGroupContext()` (line 889), and the active-context transition itself is one
atomic `compute()` call, not a separate get + conditional-remove. If that pattern was visible on an earlier revision, it looks like it has already been folded into that `compute()`.
On CI: agreed with where you and @abdessalems landed. I just re-checked - Build is still red on this head, `mergeable_state` is `blocked` on review/required-checks only (no conflicts reported), and #12311 is still open and unmerged, so the engine-v2-it diagnosis both of you gave still holds as of right now, not just as of the earlier comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Release the start latch when a blocking worker cannot start (#11679) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11727:
URL: https://github.com/apache/seatunnel/pull/11727#issuecomment-6368877597
I inspected the raw logs for the current Build linkage on head `a80834ebfb185ee0d54243e7a3ae99a5bf481817`: [run 35426701856](https://github.com/abdessalems/seatunnel/actions/runs/35426701856), attempt 1. The current failures are more specific than the earlier `SplitClusterFaultToleranceIT` explanation:
| Failed job | Direct test failure in its log |
| --- | --- |
| [engine-v2-it / JDK 8](https://github.com/abdessalems/seatunnel/actions/runs/35426701856/job/105854144544) | `BackpressureSlowSinkIT.testCheckpointsKeepCompletingUnderSustainedBackpressure`: expected at least 3 additional completed checkpoints during 90s, observed 2; samples `[1,1,1,1,1,2,3,3,3]` |
| [engine-v2-it / JDK 11](https://github.com/abdessalems/seatunnel/actions/runs/35426701856/job/105854144606) | Same backpressure failure, plus `CheckpointCoordinatorFailoverIT.testStreamJobFailsAfterCheckpointTriggerDispatchFailure`: expected `CHECKPOINT_INSIDE_ERROR`, received `Checkpoint notify complete failed` |
| [all-connectors-it-2 / JDK 8](https://github.com/abdessalems/seatunnel/actions/runs/35426701856/job/105854144530) | `OpengaussCDCIT.testAddFieldWithRestore`: assertion timeout |
| [all-connectors-it-2 / JDK 11](https://github.com/abdessalems/seatunnel/actions/runs/35426701856/job/105854144535) | Same OpenGauss restore assertion timeout |
| [all-connectors-it-7 / JDK 11](https://github.com/abdessalems/seatunnel/actions/runs/35426701856/job/105854144377) | `PostgresCDCIT.testPostgresCdcSnapshotOnlyAndCommittedOffsetStartupModes`: expected one row, observed zero, then assertion timeout |
The engine tests exercise checkpoint completion/failure semantics, so these logs alone do not establish an infrastructure-only failure or prove that #12311 resolves the current gate. They also do not establish that this PR introduced those failures. A baseline comparison or direct trace of those exact scenarios is still needed before assigning causality.
I have preserved the assertions and have not issued another workflow retry. The safe next action is to diagnose the two engine failures against this exact head and its base, then obtain passing CI on the eventual contributor revision. This is a CI-evidence update, not a new approval or a claim that the PR is ready to merge.
I also checked the concrete method names raised in the latest comment against this same fetched head:
- [`BlockingWorker.run()` lines 1285–1294](https://github.com/apache/seatunnel/blob/a80834ebfb185ee0d54243e7a3ae99a5bf481817/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/TaskExecutionService.java#L1285-L1294) already reads `taskGroupExecutionTracker.context.getClassLoaders()`. It does not resolve the class loader from the shared location-keyed map.
- There is no `finishOwnedResources()` method or its quoted monitor Javadoc in this version. [`finishExecution()` lines 1679–1701](https://github.com/apache/seatunnel/blob/a80834ebfb185ee0d54243e7a3ae99a5bf481817/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/TaskExecutionService.java#L1679-L1701) documents and performs a per-location `executionContexts.compute(...)` transition.
- [`finishExecution()` lines 1703–1728](https://github.com/apache/seatunnel/blob/a80834ebfb185ee0d54243e7a3ae99a5bf481817/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/TaskExecutionService.java#L1703-L1728) removes `cancellationFutures` using this exact context and then recycles class loaders and cancels async/timer resources for that context, including the stale-generation branch. These calls are outside a service-wide `synchronized` block.
- The test's explicitly limited deploy-return contract and separate redeploy race boundary are written at [lines 65–85](https://github.com/apache/seatunnel/blob/a80834ebfb185ee0d54243e7a3ae99a5bf481817/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/TaskDeployStaleContextRaceTest.java#L65-L85). That statement is a coverage boundary, not proof of the checkpoint CI scenarios above.
These source pointers answer those specific stale-method/context-lookup assertions; they are not an assertion that every lifecycle question or CI gate has been resolved.
<!-- daniel-fcp-repair-20261023-pr11727 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org