You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "DanielLeens (via GitHub)" <gi...@apache.org> on 2026/08/18 16:04:19 UTC
[PR] [Improve][Zeta] Log node-offline task failures as warning [seatunnel]
DanielLeens opened a new pull request, #11856:
URL: https://github.com/apache/seatunnel/pull/11856
## What changed
- downgrade expected node-offline task failures from `ERROR` to `WARN` in `PhysicalVertex`
- match only the coordinator's exact offline-node failure message
- add a focused regression test for the message classification
## Why
During scale-down, the coordinator intentionally fails task groups whose deployed worker leaves the cluster. That failure is expected, but the current log path still records it as `ERROR`, which makes routine scale-down events look like real engine faults.
## Impact
This keeps the existing failure state handling unchanged and only narrows the log level for the known node-offline case.
## Validation
- `./mvnw -nsu -Dmaven.gitcommitid.skip=true -T 3C -pl seatunnel-engine/seatunnel-engine-server -am spotless:apply`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Zeta] Log node-offline task failures as warning [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11856:
URL: https://github.com/apache/seatunnel/pull/11856#issuecomment-5555528513
CI is green on `e6e243744e3` — the fork `Build` run (33966670706) finished with all 83 non-skipped jobs passing and the `Build` check on this PR reports success.
For the record, what it took after merging `dev` (`e6e243744e3` is a pure merge commit; the three-dot diff is unchanged):
- `all-connectors-it-7` (the `PostgresCDCIT` committed-offset timeout, #11847) passed on both JDKs once the branch had #11644's larger restore budget.
- `jdbc-connectors-it-part-1 (8)` / `part-4 (8)` failed once on runner networking (TLS handshake during the Maven wrapper download; `Connection reset` from Maven Central resolving `testcontainers:postgresql`) and passed on rerun.
- `rocketmq-connector-it (8)` failed twice with `MQClientException: The broker[broker-a] not exist` / `No topic route info` across all container variants and passed on the third attempt. That signature is byte-identical to `dev`'s own run 33970546560 (`rocketmq-connector-it (11)`, 2026-09-05), and `RocketMqIT`/`RocketMqContainer` were last changed by #12050, so it looks like a dev-level intermittent failure rather than anything in this diff — worth a separate look upstream.
Review status is unchanged from the last full pass: no blocking items, only the optional Issue 5 hardening carried. @SEZ9 re-requesting your review — everything from your `CHANGES_REQUESTED` round and the later race finding is closed on this head.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Zeta] Log node-offline task failures as warning [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11856:
URL: https://github.com/apache/seatunnel/pull/11856#issuecomment-5503126335
Since this is my own PR, another self-check pass — this one scoped specifically to the commit I just pushed, `a215916e`. Nobody, including @SEZ9, has looked at it yet, so I'm not re-litigating the 8 issues from the earlier rounds; those still stand as resolved.
# What `a215916e` changes
On top of the already-reviewed marker/TTL/atomicity work, this commit:
- Removes `SeaTunnelServer`'s `GracefulShutdownAwareService` implementation and the `onShutdown()` override added in `f953140e` — the fix @SEZ9 asked for in Issue 1 of the original review, which stood unchanged for 10 days.
- Adds a hand-rolled `Runtime.getRuntime().addShutdownHook(...)` in `SeaTunnelServer.init()` (`registerShutdownHook`/`shutdownFromJvmHook`) that writes the marker and then calls `lifecycleService.shutdown()` directly.
- Sets `hazelcast.shutdownhook.enabled: false` in all 15 shipped Hazelcast config/doc locations (`config/*.yaml`, `deploy/kubernetes/**`, `docs/en|zh/**`) to stop Hazelcast's own built-in shutdown hook from running alongside the new one.
- Changes `seatunnel-cluster.sh`'s foreground path from `java ...` to `exec java ...`.
- Adds `restoringRunningJobsFromMasterSwitch` / `canClearGracefulMemberRemovalMarker()` to keep the marker alive through master-failover job recovery.
That's a much bigger footprint than a log-level fix, so I traced it against the real Hazelcast source rather than taking the new code comments at face value.
# Issue 1 (Critical) — the stated reason for the rewrite doesn't match Hazelcast's actual source, and a one-line fix was already available
**Location:** `SeaTunnelServer.java` (`registerShutdownHook`/`shutdownFromJvmHook`, new in this commit); `com.hazelcast.instance.impl.Node` and `com.hazelcast.spi.properties.ClusterProperty` in `hazelcast/hazelcast` tag `v5.1` (this project's pinned Hazelcast version — verified directly against that tag, not from memory).
The new Javadoc says: *"Hazelcast's own hook cannot provide that ordering because it puts its services into the shutting-down state before graceful-service callbacks run."* Tracing `Node` at `v5.1` directly:
```text
Node.shutdown(boolean terminate) [Node.java:502-552]
if (!terminate) callGracefulShutdownAwareServices(maxWaitSeconds) [Node.java:518-520]
-> GracefulShutdownAwareService.onShutdown() [Node.java:554-587]
// only afterward: resetJoinState(), removeShutdownHook(), shutdownServices() teardown
NodeShutdownHookThread.run() [Node.java:759-778]
switch (policy) {
case TERMINATE: hazelcastInstance.getLifecycleService().terminate(); // -> Node.shutdown(true), SKIPS graceful callbacks entirely
case GRACEFUL: hazelcastInstance.getLifecycleService().shutdown(); // -> Node.shutdown(false), same graceful path as any other caller
}
```
Hazelcast's own hook calls the *identical* `Node.shutdown(false)` method as everything else — there is no separate, earlier-teardown code path, and `GracefulShutdownAwareService` callbacks run before teardown regardless of what triggers `.shutdown()`. So "ordering" isn't actually the problem.
The real gap is one property: `ClusterProperty.SHUTDOWNHOOK_POLICY` defaults to `"TERMINATE"` (`ClusterProperty.java:1533-1534`), and this repo has never set it to `GRACEFUL` — not on `dev`, not anywhere in this PR before this commit (`git grep shutdownhook` on both only turns up `shutdownhook.enabled`, added today). So Hazelcast's own default hook, if it's what actually answers a container's SIGTERM, calls `.terminate()` — skipping `callGracefulShutdownAwareServices()` altogether. That is a real, previously-unnoticed gap in `f953140e`: all 8 prior self-review rounds and @SEZ9's review verified that `onShutdown()` runs before teardown *once invoked*, but never checked whether Hazelcast's default configuration would invoke the graceful path at all on a real signal.
Given that, the minimal fix is:
```yaml
hazelcast.shutdownhook.policy: GRACEFUL
```
added next to the existing `properties:` blocks — zero Java changes — which makes Hazelcast's own already-mature, already-self-deregistering hook call exactly the `f953140e`/`94eaf4155b` mechanism that's already been reviewed and had its race condition fixed. That keeps this PR's Java diff where it was after `94eaf4155b`, with no new hook-vs-hook interaction to reason about.
# Issue 2 (High) — the chosen fix creates a new dual-hook race for any deployment that doesn't carry the new config forward
The docs added in this commit say: *"Custom Hazelcast configurations must retain that setting to preserve this behavior."* Nothing in the code enforces or checks that. If a deployment supplies its own Hazelcast properties — an ordinary scenario for a production system customizing vendor YAML over years — without `hazelcast.shutdownhook.enabled: false`, both hooks get registered:
- Hazelcast's own `NodeShutdownHookThread` (default policy `TERMINATE`)
- SeaTunnel's new `seatunnel-graceful-member-removal-shutdown` thread
Per the JVM's own `Runtime.addShutdownHook` contract, registered hooks are started in unspecified order and run concurrently. `Node.shutdown()` has a re-entrancy guard (`setShuttingDown()`), so whichever thread's call wins that race decides graceful vs. forceful — which is scheduler-dependent, not deterministic. If Hazelcast's own hook wins, the instance tears down via `.terminate()` while SeaTunnel's hook may not have written the marker yet, reproducing the exact ERROR-instead-of-WARN symptom this PR exists to fix — silently, with no test able to catch a JVM-shutdown-hook race. That's a production-only failure mode, gated purely on whether an operator's Hazelcast config happens to carry a doc-only convention forward.
# Non-blocking, but worth recording
- `exec java ...` in `seatunnel-cluster.sh` (foreground path) is a good fix on its own merits and worth keeping regardless of how Issues 1-2 are resolved: container runtimes typically deliver SIGTERM to PID 1 only, and without `exec` the JVM here runs as a child of the launcher shell that may never see it directly.
- `restoringRunningJobsFromMasterSwitch` / `canClearGracefulMemberRemovalMarker` (retaining the marker through master-failover job recovery) reads correctly to me on a source-level pass, but it's new, untested-by-anyone-else surface added the same day as Issues 1-2, on an already large PR.
- CI on this exact head (`a215916e`) is still queued as of this review — no build/test signal either way yet.
# Recommendation
Not my call to make alone — that's the actual problem with this being round 9 of what's mostly been me reviewing my own PR. @SEZ9, given this commit swaps out the mechanism you already signed off on for a new one with the race described in Issue 2, could you take a look specifically at `a215916e` when you have a chance? I'd lean toward reverting to the `94eaf4155b` shutdown mechanism plus the one-line `hazelcast.shutdownhook.policy: GRACEFUL` config addition (Issue 1) rather than keeping the new custom hook, but that's a real design call and I don't think I should be the one deciding it on my own PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Zeta] Log node-offline task failures as warning [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11856:
URL: https://github.com/apache/seatunnel/pull/11856#issuecomment-5739315484
Hi @SEZ9, thank you for coming back to this thread, and for confirming that `dc2c64103c` closes the compile break. This is my own PR, so I am answering as the author and have re-checked everything below on the current head `dc2c64103c` (unchanged since my last comment) and in the fork CI run for that head.
## Direct answers
**F1 - where the listener is registered, where the write happens, and whether a real lifecycle path is tested**
- Registration: `SeaTunnelServer.init()` calls `getLifecycleService().addLifecycleListener(this)` at `SeaTunnelServer.java:154` (the class implements `LifecycleListener`).
- Dispatch: `stateChanged(...)` at `SeaTunnelServer.java:239-245` sends `SHUTTING_DOWN` to `markLocalGracefulMemberRemoval()` (`:284-286`) and `STARTED` to the clear (`:292-294`).
- The IMap write is `gracefulMemberRemovalIMap.put(thisAddress, System.currentTimeMillis(), TTL, MILLISECONDS)` at `SeaTunnelServer.java:309-313`, inside `updateLocalGracefulMemberRemovalMarker` (`:300-326`). `ManagedService.shutdown()` (`:247-278`) no longer touches the map, and `SeaTunnelServerShutdownTest.shouldNotMarkGracefulMemberRemovalDuringManagedServiceShutdown` pins that.
- Real lifecycle test: `ClusterFaultToleranceIT.testGracefulShutdownPublishesMemberRemovalMarker` (`ClusterFaultToleranceIT.java:79-146`) starts two real members with `SeaTunnelServerStarter.createHazelcastInstance`, registers an entry listener on node2's real map proxy (`:108-119`), calls `node1.shutdown()` (`:121`), and asserts the marker arrives and is accepted by `CoordinatorService.isGracefulMemberRemovalMarkerValid` (`:124-131`). The mock-based `SeaTunnelServerShutdownTest` only covers the branch logic.
- Evidence that it ran: in fork run `35240527669` (this head, attempt 3), the `engine-v2-it (8, ubuntu-latest)` log reports `Tests run: 10, Failures: 0, Errors: 0, Skipped: 2 - in ClusterFaultToleranceIT`. The two skips match the two `@Disabled` tests that already exist in that file on `dev` and on this head. That the new test itself passed is an inference from the class-level result; I did not find a per-method line for it.
- What that test does not cover: it exercises `HazelcastInstanceImpl.shutdown()`, not the JVM shutdown-hook thread (SIGTERM) path. That the hook reaches the same `LifecycleServiceImpl.shutdown` and fires `SHUTTING_DOWN` is from my reading of the Hazelcast 5.1 sources, not from a test in this PR.
**F4 - TTL versus timestamp**
Both, with different jobs. Eviction is a Hazelcast-side per-entry TTL: the 4-argument `put(key, value, ttl, unit)` at `SeaTunnelServer.java:309-313` with `Constant.GRACEFUL_MEMBER_REMOVAL_MARK_TTL_MILLIS` = 5 minutes (`Constant.java:72`). Separately, `CoordinatorService.isGracefulMemberRemovalMarkerValid` (`CoordinatorService.java:2157-2161`) checks `abs(now - markedAt) <= 5 min`. That second check does compare the departing member's `System.currentTimeMillis()` with the coordinator's clock, so it tolerates up to 5 minutes of skew; beyond that the marker is rejected and the failure stays at `ERROR`, which is the safe direction. So the "no eviction" part of F4 is resolved by the native TTL, and the timestamp is only a secondary guard.
**F2 - destructive removal and failover**
The read is non-destructive: `gracefulMemberRemovalIMap.get(lostAddress)` at `CoordinatorService.java:2185`. The clear runs only after the propagation loop (`:2222-2238`), and only when no master-switch restore is in flight (`canClearGracefulMemberRemovalMarker`, `:2169-2172`, called at `:2239-2245`). It is a value-conditional `remove(lostAddress, markedAt)` (`:2207`), so it cannot delete a newer marker from a restarted member at the same address. When the clear is skipped, or a coordinator dies mid-processing, the entry stays in the IMap until its TTL, and the new master classifies through `PhysicalVertex.checkTaskGroupIsExecuting` (`PhysicalVertex.java:239-250`), which reads the same marker without clearing it (`:278-288`) and applies the same helpers (`recordMemberRemovedFailure`, `:760-770`). Unit tests: `shouldRetainGracefulMemberRemovalMarkerDuringMasterSwitchRecovery` and `shouldClassifyMissingWorkerDuringMasterFailoverAsGraceful`. One nit I noticed while re-reading: the Ja
vadoc at `PhysicalVertex.java:273-277` still says "one-time" marker, which is stale wording now that the read is non-destructive.
## Per-finding status on `dc2c64103c`
| Finding | Status | Evidence |
|---|---|---|
| F1 marker written while PASSIVE | Fixed in source; publication exercised by a real two-node IT that passed on this head | `SeaTunnelServer.java:154`, `:239-245`, `:309-313`; `ClusterFaultToleranceIT.java:79-146` |
| F2 consumed before processing / failover | Fixed | `CoordinatorService.java:2185`, `:2207`, `:2239-2245`; `PhysicalVertex.java:278-288` |
| F3/F6 `JobException` replaced by a string | Fixed; payload shape unchanged | `buildMemberRemovedFailureState`, `CoordinatorService.java:2144-2149`; `shouldKeepThrowablePayloadForMemberRemovedFailureState` |
| F4 eviction | Fixed by native TTL; timestamp is a secondary guard | `SeaTunnelServer.java:309-313`; `CoordinatorService.java:2157-2161` |
| F5/F8 message matching and duplicated template | Fixed | no `Pattern` or `.matches(` in `PhysicalVertex.java`; single producer `buildMemberRemovedOfflineMessage` (`CoordinatorService.java:2133-2137`), called from `:2146` and `PhysicalVertex.java:768`; the classification is a typed boolean |
| F7 docs | Present | `docs/en/engines/zeta/state-storage-and-recovery.md` (new section at line 155) and the zh counterpart |
## What CI actually shows on this head
I read fork run `35240527669` (attempt 3, finished 2026-09-18T14:13Z; the apache-side `Build` check finished as failure at 14:14Z). It is red, so I cannot call this PR verified or ready to merge yet.
- The compile break is fixed: `unit-test` passed on Java 8 and 11 on both ubuntu and windows, and `engine-v2-it (11)` passed. 79 of 94 jobs succeeded, 11 were skipped, 3 failed, and 1 was cancelled.
- `engine-v2-it (8)` failed with 204 tests run, 1 failure and 1 error: `BackpressureSlowSinkIT.testCheckpointsKeepCompletingUnderSustainedBackpressure` (only 1 of the expected 3 additional checkpoints), and `SplitClusterFaultToleranceIT.testStreamJobCancelResolvesWhenWorkerCrashesBeforeCancelAck` (`expected: <CANCELED> but was: <FAILED>`). Both match the targets of open PRs #12316 and #12311, so a `dev` sync would not clear them. My diff does not touch the vertex-state filter or the FAILED target in `makeTasksFailed` (`CoordinatorService.java:2256-2265`); it only adds the classification argument. That these two are unrelated to my change is my inference, not something a run has proven.
- `all-connectors-it-2 (8)` and `(11)` both failed on `OpengaussCDCIT.testAddFieldWithRestore` (`ConditionTimeout` at `:476`). This is a CDC connector test unrelated to the engine shutdown path, as far as I can tell.
- `jdbc-connectors-it-part-1 (11)` was cancelled at 14:13Z after running about two hours, with no failed test in its log. I have not established why it was cancelled.
## Runtime evidence for the WARN downgrade (the acceptance list from my last comment)
Counts below are from the `engine-v2-it (8)` log only, which is one job of the run.
- `... end with state FAILED due to node offline` at `WARN`: 125 occurrences (baseline on the previous head: 0). No `ERROR`-level offline line remains in that log.
- `NoDataMemberInClusterException`: 0 occurrences.
- `Failed to clear graceful member removal marker`: 13, all `HazelcastInstanceNotActiveException`, logged from the coordinator's clear (`CoordinatorService.java:2207`). This is not fully gone; it fires when the coordinator itself is stopping.
- `Failed to mark graceful member removal marker`: 39, all `HazelcastInstanceNotActiveException`, so the criterion "gone for shutdown() paths" is not literally met. I traced four of them in `ClusterFaultToleranceIT`, plus one in `SplitClusterPendingJobLifecycleFailoverIT`, and they are calls to `shutdown()` on an instance that was already down: for example `testBatchJobRestoreIn2NodeWorkerDown` shuts node2 down in the body (`:441`) and again in `finally` (`:470`), and the same pattern holds at `:540`/`:591`, `:660`/`:690` and `:763`/`:809`. `LifecycleServiceImpl.shutdown` fires `SHUTTING_DOWN` on every call, so the listener tries a write against a dead instance. I have not traced the other 34 individually, so I cannot claim they are all of this kind.
## Remaining gaps, stated plainly
1. CI on this head is red (list above). I am not treating the PR as ready until the failed jobs are re-run and pass, or are shown to be unrelated.
2. No test asserts the `WARN` versus `ERROR` classification in a real cluster. The engine-level evidence is the log counts above; the classification rules themselves are covered by unit tests only (`PhysicalVertexTest`, `CoordinatorServiceMemberRemovedTest`).
3. My earlier recommendation N2 is still open on this head: the marker `put` (`SeaTunnelServer.java:309`) and the `STARTED` clear (`:315`) are synchronous calls on lifecycle threads. Together with the stack-trace noise above, a bounded async call, or skipping the write when the instance is already not running, would be a reasonable follow-up.
4. N4 (one sentence in the docs about the 5-minute TTL versus `hazelcast.max.no.heartbeat.seconds`) is also still open.
I will do a final pass on the next green run of this head. Thanks again for the careful review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Zeta] Log node-offline task failures as warning [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11856:
URL: https://github.com/apache/seatunnel/pull/11856#issuecomment-5825959553
Thanks for the detailed follow-up and for re-checking the review body via the API — agreed the cut-off looks like a rendering artifact rather than a content gap. Consolidating where things stand:
**F1 – marker published from `ManagedService.shutdown()`**
- Agreed the Hazelcast-version dependency note for `markLocalGracefulMemberRemoval()` is a recommended fix not yet landed (`dc2c64103c` is unchanged). Please add a short comment near that method in `SeaTunnelServer.java` stating that correctness relies on `LifecycleServiceImpl.shutdown()` firing `SHUTTING_DOWN` before distributed-object proxies stop being served, and noting the Hazelcast version this was verified against.
- Test coverage: class-level `Tests run: 10, Failures: 0, Errors: 0, Skipped: 2` for `ClusterFaultToleranceIT` from run `35240527669` is noted. A method-level result for `testGracefulShutdownPublishesMemberRemovalMarker` is still unconfirmed — please attach the per-method surefire line for it and confirm it is not one of the 2 skipped tests.
**F2 – destructive consume / no physical expiry**
The `CoordinatorService.java:2185` explanation was cut off mid-sentence in the thread, so I can't act on it yet. Could you restate briefly: (a) whether the marker is read non-destructively before task-failure processing and when it is cleared, and (b) the intended behavior if the master fails over between the read and the clear?
**F3–F8**
Since `dc2c64103c` is unchanged, these remain open as written in the previous review (String vs `JobException` payload in `TaskExecutionState`, eviction for `engine_gracefulMemberRemoval`, message-pattern-based WARN downgrade in `PhysicalVertex`, persisted error payload change, docs, and the duplicated offline-message template). A quick note on which you plan to address in the next push versus defer would help me track the thread.
Once the new commit is up I'll re-review against these points only.
<!-- streview-comment:1304 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Zeta] Log node-offline task failures as warning [seatunnel]
Posted by "SEZ9 (via GitHub) via commits" <co...@seatunnel.apache.org>.
SEZ9 commented on PR #11856:
URL: https://github.com/apache/seatunnel/pull/11856#issuecomment-6889026742
Thanks for the follow-up on `1f003592570833eb48b2a7edad0830e960e1c462`. Fixing the regression test from `b32e5360d88` to import the concrete type returned by `MembershipServiceEvent#getMember()` looks like the right correction, and keeping the mock and assertions unchanged is fine.
One note: this head only touches the test import, so none of the points from the earlier review are addressed yet. Still open:
1. **Marker publish timing (HIGH)** – the graceful-shutdown marker is written from `ManagedService.shutdown()`, when the Hazelcast node is already PASSIVE and IMap writes can be rejected. Please move the publish to a point where the member is still ACTIVE (or show evidence that the write succeeds in a real graceful shutdown), otherwise the WARN downgrade can silently no-op.
2. **Marker lifecycle in `CoordinatorService`** – the marker is destructively consumed before task-failure processing, so a master failover mid-processing loses the classification. Please consume after processing (or make the read non-destructive) and give the entries a real expiry.
3. **Error payload on the graceful path** – replacing the `JobException` in `TaskExecutionState` with a plain `String` changes what reaches job history / REST error fields and contradicts the "state handling unchanged" guarantee. Please keep the throwable and only adjust the log level.
4. **`engine_gracefulMemberRemoval` eviction** – no TTL/eviction configured and the manual TTL check relies on cross-node clocks. Please configure map-level TTL or max-idle instead.
5. **Classification by message matching in `PhysicalVertex`** – deciding ERROR vs WARN by regex over free-text lets any failure that mimics the offline template be downgraded. Please carry an explicit flag/typed cause rather than matching text.
6. **Duplicate template** – the offline message is a format string in `CoordinatorService` and a regex in `PhysicalVertex`; please share a single constant (or, better, drop the regex per point 5).
7. **Docs** – please add a short section under `docs/` describing the graceful-shutdown marker map and the WARN downgrade behavior.
Also, since no local compile or runtime validation was run on this head, please let the GitHub CI run complete and share the unit and engine integration results here before we call the test fix verified.
<!-- streview-comment:2364 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Zeta] Log node-offline task failures as warning [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11856:
URL: https://github.com/apache/seatunnel/pull/11856#issuecomment-5549171990
Pushed `9e433426e49` addressing the open items from the last full re-review (the two Medium findings plus the carried-over Low ones), and closing out the CI question on the previous head.
**Issue 1 (Medium) — duplicated offline-message template / TTL rule.** `CoordinatorService.buildMemberRemovedOfflineMessage` and `isGracefulMemberRemovalMarkerValid` are now `public static` and are the single source of truth; `PhysicalVertex.recordMemberRemovedFailure` delegates to both instead of re-deriving the string and the `Math.abs(...) <= TTL` check. No behavior change — both copies computed the same thing — but the master-failover restore path can no longer drift from the membership-callback path. `PhysicalVertexTest.shouldAgreeWithCoordinatorRulesAtMarkerTtlBoundaries` pins the two consumers together at the TTL boundaries (absent / fresh / exactly-at-TTL / one-past-TTL on both sides), so an edit to either side alone now fails a test.
**Issue 2 (Medium) — restore-path integration had no test.** `checkTaskGroupIsExecuting` is now package-visible (`@VisibleForTesting`, no logic change) and three new tests drive it against mocked `NodeEngine` / `HazelcastInstance` / `ClusterService` / `IMap`, with the task recorded as `RUNNING`, its owned slot pointing at a worker that is no longer a cluster member:
- fresh marker present → the vertex reads `engine_gracefulMemberRemoval` keyed by the worker address, records the shared offline message classified graceful, returns `false`, and never touches the operation service;
- no marker → same message, classified unproven (ERROR path, identical to `dev`);
- marker read throws → exception swallowed, still classified unproven (fails safe toward ERROR).
The map-name and key-type wiring the review flagged as unverifiable is now exercised, not just the pure helper.
**Issue 3 (Low) — single-line `/** ... */` Javadocs.** This turned out to be formatter-mandated rather than a slip: the repo runs google-java-format 1.7 (AOSP) through Spotless with no Javadoc opt-out, and GJF collapses any Javadoc whose text fits within the line limit back to the single-line form — I hand-wrapped all of them and `spotless:apply` immediately re-collapsed the short ones, so the `Code style` job would reject a hand-wrapped version. `docs/en/developer/coding-guide.md` has no guidance on Javadoc form either. What I did instead: where a one-liner was genuinely under-explaining its constraint (why the marker is read without clearing, why a stale marker must be cleared, why only the graceful classification downgrades the level, why the message and flag are held together), the comment now says so and naturally spans multiple lines; the remaining short ones stay in the form the formatter produces.
**Issue 4 (Low) — `.exceptionally()` branch untested.** `SeaTunnelServerShutdownTest.shouldAbsorbFailedAsyncMarkerClearWithoutBlockingStartup` returns a pending future from `removeAsync`, asserts a completion handler is chained before completion (i.e. the clear is not fire-and-forget), then fails the future and asserts nothing escapes the lifecycle callback and no blocking `remove` fallback occurs. The warning text itself is not asserted (static Hazelcast logger).
**Issue 5 (Low, informational)** — intentionally unchanged: it fails safe toward ERROR and the review marked it optional.
**CI on the previous head (`c646f4cd8`):** the Build workflow finished 78 pass / 10 skipped / 1 fail. The one failure was `PostgresCDCIT.testPostgresCdcSnapshotOnlyAndCommittedOffsetStartupModes` (`ConditionTimeoutException`, expected 1 row got 0 after committed-offset restore) in `connector-cdc-postgres-e2e` — a module this Zeta-only diff does not touch. That exact symptom is the open upstream Postgres-CDC bug #11847 (fix in flight in #11864), and the suite has a history of timing flakes (#11644, #11202). Rather than rerun that job in isolation I let this push start a fresh full run; if only that flake recurs I'll rerun it at job granularity.
Local verification was limited to module-scoped `spotless:apply`; compile and tests run on GitHub CI against this head.
@SEZ9 with this the Medium findings from the latest re-review are closed on the current head — whenever you have time, a fresh look would be appreciated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Zeta] Log node-offline task failures as warning [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11856:
URL: https://github.com/apache/seatunnel/pull/11856#issuecomment-5852104617
Thanks for the follow-up — head is still `dc2c64103c`, so I'm treating everything below as pending the next push.
**F1**
- Comment in `markLocalGracefulMemberRemoval()` (`SeaTunnelServer.java`): sounds right. Please make sure it spells out the dependency on `LifecycleServiceImpl.shutdown()` firing `SHUTTING_DOWN` before distributed-object proxies stop being served, and the Hazelcast version (5.1) it was verified against.
- Per-method result: agreed that the class-level `Tests run: 10, Failures: 0, Errors: 0, Skipped: 2` from run `35240527669` isn't enough on its own. Once the new run is in, please paste the surefire line for `testGracefulShutdownPublishesMemberRemovalMarker` here so we can confirm it actually executed and isn't one of the two skips. F1 stays open until then.
**F2**
- The (a) walk-through — non-destructive `get` at `CoordinatorService.java:2185`, value-conditional `remove` at `:2207` only after the propagation loop (`:2222-2238`) and only when `canClearGracefulMemberRemovalMarker` allows it (`:2239-2245`) — addresses the "destructively consumed before processing" concern for the event that reads the marker. Thanks for restating it in full.
- (b) failover: the point that the new master re-derives the classification via `PhysicalVertex.checkTaskGroupIsExecuting` (`PhysicalVertex.java:278-288`) without clearing is a reasonable answer. Please add a short comment at that read site noting it is intentionally non-clearing and why, so the two read paths don't drift apart later.
- The `restoringRunningJobsFromMasterSwitch` reset bug you found is exactly the kind of thing that turns the explicit clear into dead code and pushes every removal onto the TTL — glad you're treating it as blocking. For the fix, please: (1) reset the flag when the restore actually completes (including the failure/early-exit paths of the restore, not just the happy path), and (2) add a test that performs a master switch with active jobs and then verifies a subsequent graceful removal clears the marker immediately rather than waiting out the TTL. That test is what will let us close F2.
**Remaining asks for the next push**
1. F1 code comment + per-method surefire line for `testGracefulShutdownPublishesMemberRemovalMarker`.
2. F2 flag-reset fix + master-switch regression test, plus the non-clearing comment on the `PhysicalVertex` read.
3. A status line on F3–F8 (throwable vs. string payload, marker eviction/TTL, message-based classification, docs, and the duplicated offline template) — even a "not yet addressed" is fine, I just want them tracked alongside F1/F2 rather than falling off.
Ping when the push is up and I'll re-review against the new head.
<!-- streview-comment:1354 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Zeta] Log node-offline task failures as warning [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11856:
URL: https://github.com/apache/seatunnel/pull/11856#issuecomment-5426486083
Pushed the fix for the race in `94eaf415`, going with **Option A** from the discussion above so @SEZ9 doesn't need to build the follow-up commit.
What changed:
- `PhysicalVertex` now records the failure message and the coordinator-classified graceful member-removal flag as one immutable `FailureClassification` holder, installed by a single first-write-wins `compareAndSet` — the mismatched `compareAndSet`/`set` pair on two independent atomics is gone. Both writer paths (`updateStateByExecutionService` and `makeTaskGroupFailing`) go through the same install method, and the FAILED branch of `stateProcess()` reads the holder once, so the message and its classification can never be observed as a torn pair.
- One deliberate nuance to preserve the pre-existing semantics exactly: a `null` message never claims the slot (the old `compareAndSet(null, null)` "succeeded" but left the reference empty, so a later real message could still win it); an empty-string message still claims the slot, as before. The graceful=true path always carries a real message (`buildMemberRemovedFailureState` wraps `JobException`), so WARN classification for genuine graceful removals is unaffected, and every degenerate path falls toward ERROR — the safe direction.
Tests added in `PhysicalVertexTest`:
- `shouldNotRetagRecordedGenuineFailureAsGraceful` — deterministically reproduces the exact interleaving from the review (genuine failure records first, node-offline classification lands after): the recorded pair must stay `{genuine message, graceful=false}`. On the previous head this same sequence ended with `{genuine message, graceful=true}`, i.e. the WARN misclassification.
- `shouldNotStripGracefulFlagFromRecordedOfflineFailure` — the reverse order stays `{offline message, graceful=true}`, matching first-write-wins.
- `shouldIgnoreNullMessageAndKeepSlotClaimable` — the null-message semantics above.
- `shouldKeepMessageAndFlagPairedUnderConcurrentWriters` — two latch-released writer threads, 100 iterations, order-independent assertion: the recorded flag always belongs to whichever message won the slot.
Verification: ran module-scoped `spotless:apply`/`spotless:check` locally (clean); compile/test runs on GitHub CI against this head (`94eaf415`). The previous run's only failure was `RocketMqIT.testSourceRocketMqRestore` in `rocketmq-connector-it` (message-count flake preceded by a `RemotingSendRequestException` against the test container) — unrelated to this engine-only diff. I'll track the new run and retry at job granularity if only that flake reappears.
@SEZ9 with this, the 8 findings from your `CHANGES_REQUESTED` review and the race flagged afterwards are all addressed on the current head — please take a fresh look when you get a chance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Zeta] Log node-offline task failures as warning [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11856:
URL: https://github.com/apache/seatunnel/pull/11856#issuecomment-5342958510
Thanks for working on this! I re-traced the whole call chain independently against the current head (`51702fa2b`) before reading back through the thread, so let me fold my pass into the review already posted above rather than duplicate it — the conclusion is the same, and I want to add two things that review flagged as open: extra source-level confirmation for Issue 1, and the CI root cause (the earlier pass noted "logs were not retrievable at review time").
# Confirmation of Issue 1 (Critical) — the regex is unreachable on the real path
Tracing `errorByPhysicalVertex` end to end:
```
CoordinatorService.makeTasksFailed() [CoordinatorService.java:2006-2013]
-> new TaskExecutionState(loc, FAILED, new JobException(String.format(
"The taskGroup(%s) deployed node(%s) offline", taskGroupLocation, lostAddress)))
-> TaskExecutionState(TaskGroupLocation, ExecutionState, Throwable) [TaskExecutionState.java:32-40]
throwableMsg = ExceptionUtils.getMessage(throwable)
-> ExceptionUtils.getMessage(Throwable) [ExceptionUtils.java:28-42]
calls throwable.printStackTrace(pw) and returns the full dump, NOT throwable.getMessage()
```
`JobException` (`org.apache.seatunnel.engine.common.exception.JobException`) has no `toString()` override, so `Throwable.toString()` falls back to `getClass().getName() + ": " + getMessage()`. That means the first line `printStackTrace()` emits is literally:
```
org.apache.seatunnel.engine.common.exception.JobException: The taskGroup(...) deployed node(...) offline
at org.apache.seatunnel.engine.server.CoordinatorService.lambda$makeTasksFailed$...(CoordinatorService.java:2011)
at java.util.ArrayList.forEach(ArrayList.java:...)
...
```
`DEPLOYED_NODE_OFFLINE_ERROR_PATTERN` (`PhysicalVertex.java:81-82`) is anchored `^The taskGroup\(.+\) deployed node\(.+\) offline$` and checked with `.matches()` — a whole-string match, no `DOTALL`/`MULTILINE`. The class-name prefix alone fails the `^The taskGroup` anchor, independent of the multi-line issue. So `isDeployedNodeOfflineFailure()` returns `false` for every real invocation, `stateProcess()`'s `FAILED` branch always falls through to the `else`, and `log.error` still runs unconditionally — this PR ships with zero behavior change on its only production call path. `makeTaskGroupFailing()` (`PhysicalVertex.java:654-656`) has the identical `ExceptionUtils.getMessage()` write pattern, so there's no alternate producer that would store a bare message either.
The new `PhysicalVertexTest` only calls the static predicate with a hand-written clean string, so it can't catch this — it never goes through `TaskExecutionState(..., Throwable)` the way `CoordinatorService` actually does.
I'll also add one adjacent observation on scope, since it affects how Issue 2 should be fixed: `SubPlan.addPhysicalVertexCallBack` (`SubPlan.java:209-219`) already logs `log.error("Task %s Failed in %s, Begin to cancel other tasks in this pipeline.", ...)` unconditionally for **every** `FAILED` task, node-offline or not, before the pipeline decides whether to auto-restore from checkpoint (`SubPlan.java:737-744`, gated by `checkNeedRestore`/`pipelineMaxRestoreNum`). So even once Issues 1 and 2 are fixed, the pipeline-level ERROR signal that "a task failed" will still fire for node-offline events — operators/alerting scraping for ERROR won't lose visibility that *something* failed. What they will lose is the node-address detail this PR's WARN message carries, since that detail only exists in the (currently dead) `PhysicalVertex`-level log line. Worth keeping in mind when deciding whether to also downgrade or restructure the `SubPlan`-level log once Issue 2's crash/scale-down disti
nction is designed.
# CI diagnosis (root cause now confirmed, both unrelated to this diff)
Fork run [`32158223580`](https://github.com/DanielLeens/seatunnel/actions/runs/32158223580) for this exact head (`51702fa2b454ce02b38409db7caec76ee0e1aa90`) has 2 failing jobs (not 3 — `all-connectors-it-7` isn't part of this run's job list at all):
- **`unit-test (11, windows-latest)`** — [job log](https://github.com/DanielLeens/seatunnel/actions/runs/32158223580/job/95784074576): `JobStateEventTest>AbstractSeaTunnelServerTest.before:70 » IllegalState Node failed to start`. This is the known Windows/Hazelcast node-bootstrap flake seen repeatedly on `windows-latest` runners across unrelated PRs — environmental, not connected to `seatunnel-engine-server`'s `dag/physical` package this PR touches.
- **`engine-v2-it (11, ubuntu-latest)`** — [job log](https://github.com/DanielLeens/seatunnel/actions/runs/32158223580/job/95784074646): `ClusterFaultToleranceTwoPipelineIT.testTwoPipelineStreamJobRestoreIn2NodeMasterDown` fails with `SeaTunnelEngineRetryableException: Can not get coordinator service from an active master node.`, thrown from `SeaTunnelServer.java:313` — a bounded-retry timing window during master failover election in a 2-node kill-the-master test (202s test, 17s to this specific failure). This exception path is in `SeaTunnelServer`/master-election, not anywhere this PR's diff touches (`PhysicalVertex.stateProcess()`'s log-level branch and the new regex helper), so it reads as pre-existing CI timing flakiness for this specific IT rather than something introduced here.
Neither failure is caused by this diff. They should still be re-run/confirmed green (or explicitly triaged as known flakes) before merge, but they are not evidence against the change itself — the blocking issues are the ones above.
# Merge Recommendation (unchanged)
### Conclusion: Not recommended for merge
Blockers (carried over from the review above, now with added evidence):
1. Issue 1 (Critical) — confirmed via `JobException`/`ExceptionUtils.getMessage()` source: the WARN downgrade never fires in production; needs a structural signal (typed exception / flag on `TaskExecutionState`) set at `CoordinatorService` and consumed as-is, not regex-matching a rendered stack trace.
2. Issue 2 (High) — the trigger (`MembershipServiceEvent` member-removed) doesn't distinguish graceful scale-down from crash/OOM-kill/network-partition, so a correct fix for Issue 1 would also silence genuine outages at WARN unless scope is narrowed.
3. CI — both current failures are environmental/timing flakes unrelated to this diff (see above); not a code blocker, but should be green (or explicitly waived as known flake) before merge.
No blocking issue with the new test's stability or license/formatting — those are fine as-is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Zeta] Log node-offline task failures as warning [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11856:
URL: https://github.com/apache/seatunnel/pull/11856#issuecomment-5711741991
[REVIEW] Maintainer re-review of head `b91afe70e1`, two commits after the head I reviewed yesterday (`fd60dfd596`): `f3308d8168` "Correct graceful member removal lifecycle" and `b91afe70e1` "Remove no-op files from PR diff". I re-read the full diff against `dev` (now 14 files, +1245/-34) and the incremental diff, and I read the CI logs of the run on this head rather than its status only.
**Verdict: not ready to merge. The design fixes for both P0s from the last round are correct at source level, but this head does not compile its test sources, so CI is red on 65 jobs and the fix has no runtime verification at all.** The Validation section of the PR body claims "real two-node engine E2E coverage proving that `SHUTTING_DOWN` publishes a marker". On this head that test has never executed.
# 1. Blocking: the head does not compile (N1)
```text
[ERROR] COMPILATION ERROR :
[ERROR] .../seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/SeaTunnelServerShutdownTest.java:[72,27] error: cannot find symbol
symbol: variable TimeUnit
location: class SeaTunnelServerShutdownTest
[ERROR] Failed to execute goal ...maven-compiler-plugin:3.10.1:testCompile (default-testCompile) on project seatunnel-engine-server
```
- Cause: `f3308d8168` removed `import java.util.concurrent.TimeUnit;` from `SeaTunnelServerShutdownTest` while line 72 still asserts `eq(TimeUnit.MILLISECONDS)` on the marker `put`. The fix is to restore that one import.
- Scope: fork run `35175217738` on this head has 65 failed jobs. I opened five of them (`unit-test (8)`, `engine-v2-it (8)`, `kafka-connector-it (8)`, `jdbc-connectors-it-part-1 (8)`, `transform-v2-it-part-1 (11)`); all five stop at this exact error. The 9 green jobs are the ones that never compile Java test sources (license, dead links, code style, Helm, website, UI, dependency licenses, sanity, changes). Two `oracle-cdc` jobs were still running and the apache-side `Build` check was pending when I looked.
- Consequence: `ClusterFaultToleranceIT.testGracefulShutdownPublishesMemberRemovalMarker`, the new unit tests, and every existing fault-tolerance IT did not run. The local validation listed in the PR body (scoped spotless plus `git diff --check`) cannot catch a missing import, so please do not describe the E2E as "proving" anything until a CI run has executed it.
# 2. Status of yesterday's findings on this head
| # | Finding from the last round | Status on `b91afe70e1` | Evidence |
|---|---|---|---|
| 1 (P0) | Marker write from `GracefulShutdownAwareService.onShutdown()` can never succeed | **Fixed in source, unverified at runtime.** `GracefulShutdownAwareService` is gone; `stateChanged(SHUTTING_DOWN)` now writes the marker | `SeaTunnelServer.java:239-245`, `:300-326`; listener registered at `:154`. `LifecycleServiceImpl.shutdown()` fires `SHUTTING_DOWN` at `:94` before `node.shutdown()` at `:101`, so `Node.isRunning()` is still `true` there |
| 2 (P0) | Unguarded `IMap.get` ahead of the failure-propagation loop | **Fixed.** Read and clear are wrapped and fall back to the unproven/`ERROR` path | `CoordinatorService.java:2180-2195` (read), `:2202-2215` (clear); tests `shouldTreatMarkerReadFailureAsUnproven`, `shouldAbsorbMarkerClearFailure` |
| 3 | `hazelcast.shutdownhook.policy: GRACEFUL` changed SIGTERM semantics everywhere | **Resolved.** All 17 config/doc/k8s-conf files are byte-identical to the merge base again; SIGTERM behaves exactly as on `dev` | `git diff --quiet <merge-base> HEAD -- config deploy/.../conf ...` reports identical |
| 4 | Startup clear at `STARTING` failed on every lite-member worker boot | **Fixed.** Clear moved to `STARTED`, which `HazelcastInstanceFactory.java:241` fires only after the join | `SeaTunnelServer.java:242-244` |
| 5 | Cancel path now records an "offline" message | **Acknowledged** in the PR body under "Behavior and compatibility" | PR description |
| 6 | Javadoc/docs described the wrong ordering | **Fixed.** Javadoc and both `state-storage-and-recovery.md` pages now describe `SHUTTING_DOWN` and the default `TERMINATE` policy | docs diff |
| 7 | No real-cluster test | **Added, never executed** (see N1) | `ClusterFaultToleranceIT.testGracefulShutdownPublishesMemberRemovalMarker` |
One improvement beyond what I asked for: the coordinator clear is now value-conditional, `remove(lostAddress, markedAt)` at `CoordinatorService.java:2207`, so it cannot delete a newer marker written by a restarted member at the same address. Good change.
I also checked the claim in the new docs and PR body that only intentional exits publish a marker. It holds in Hazelcast 5.1: the callers that go through `LifecycleService` are `NodeShutdownHookThread` (`Node.java:766,769`), `HazelcastInstanceImpl.shutdown()` (`:346`), the factory's `shutdownAll`/`terminateAll`, and the operator cluster-shutdown API (`ClusterServiceImpl.java:959,986`, `ShutdownNodeOp.java:44`). Fault paths bypass it and call `node.shutdown(true)` directly, so they emit no `SHUTTING_DOWN`: `OutOfMemoryHandlerHelper.tryShutdown` (`:62`, which is where SeaTunnel's `ExceptionUtil` routes an `OutOfMemoryError` via `OutOfMemoryErrorDispatcher`) and `ClusterServiceImpl.java:400`. A JVM OOM therefore stays at `ERROR`, as it should.
# 3. New non-blocking findings
**N2 (Medium, recommended): both lifecycle hooks now do synchronous distributed calls on lifecycle threads.**
- `stateChanged(SHUTTING_DOWN)` runs a blocking `IMap.put` inside `LifecycleServiceImpl.shutdown()`'s `synchronized (lifecycleLock)` block (`LifecycleServiceImpl.java:93-94`), on the caller's thread. Under the default `TERMINATE` policy that caller is the JVM shutdown-hook thread, in front of a `terminate()` that is expected to be immediate. `stateChanged(STARTED)` runs a blocking `IMap.remove` on the thread that constructs the instance.
- Both are bounded, and both fail safe. With the shipped `hazelcast.invocation.max.retry.count: 20`, retryable errors give up after roughly 3.5 s (`Invocation.handleRetry`, `Invocation.java:706-712`). The long tail is a partition owner that is alive but unresponsive: that waits for the operation call timeout, which is the same ~120 s stall this PR already hit once at `45e981d3`.
- This is an inference from the source, not something I observed. Operators restart nodes precisely when the cluster is degraded, so I would not let a best-effort log hint hold the shutdown hook for two minutes. Suggestion: `putAsync(...)` / `removeAsync(...)` with a short bounded wait of a few seconds, keeping the existing catch-and-WARN fallback. On a full-cluster stop some nodes will also log `Failed to mark graceful member removal marker` with a stack trace because their peers are already gone; consider dropping the stack trace for that known case.
**N3 (Low): the new E2E proves publication, not classification.** It shuts down `node1` with no job running and asserts that `node2` sees the marker through a real proxy. That is the right proof for the defect that was actually broken, and using an entry listener instead of polling `get` is correct, because the surviving coordinator clears the marker right after `memberRemoved`. The end-to-end evidence that the log level really narrows should come from the next engine job log, where the existing worker-down ITs already produce these lines. Acceptance criteria I will check myself on the next green run, against yesterday's baseline (153 write failures, 0 WARN, 122 ERROR in `engine-v2-it (8)`):
- `Failed to mark graceful member removal marker` followed by `HazelcastInstanceNotActiveException` is gone for `shutdown()` paths;
- `end with state FAILED due to node offline` (WARN) appears for the worker-down tests;
- `Failed to clear graceful member removal marker` with `NoDataMemberInClusterException` is gone.
**N4 (Low, docs): marker TTL versus failure detection.** The TTL is 300 s (`Constant.java:72`) and the shipped configs detect a silent member after `hazelcast.max.no.heartbeat.seconds: 180`. An operator who raises that timeout above 300 s will see heartbeat-detected departures classified as `ERROR` because the marker expires first. It fails safe, so one sentence in `state-storage-and-recovery.md` is enough.
# 4. @SEZ9, on your comment from 02:55Z
Your comment was written against `fd60dfd596` and lists F1 plus F2 through F8. F2 through F8 were closed in earlier rounds; I re-verified each on this head so the thread has one clear status:
- **F3/F6 payload:** `buildMemberRemovedFailureState` wraps the message in `JobException` on both paths (`CoordinatorService.java:2144-2149`), pinned by `shouldKeepThrowablePayloadForMemberRemovedFailureState`. No plain-string branch exists any more.
- **F2/F4 marker lifecycle:** the read is a non-destructive `get` (`:2185`); the clear runs after the propagation loop, only when no master-switch restore is in flight, and is value-conditional (`:2207`); the write uses Hazelcast's native TTL overload (`SeaTunnelServer.java:309-313`). The timestamp check is a secondary guard with a ±5 min skew tolerance that fails toward `ERROR`.
- **F5/F8 message matching:** there is no regex left in `PhysicalVertex` (no `Pattern`, no `.matches(`). Classification is a typed boolean carried through `updateStateByExecutionService(state, graceful)` and the immutable `FailureClassification` holder, and the offline message has a single producer, `CoordinatorService.buildMemberRemovedOfflineMessage`.
- **F7 docs:** present in `docs/en|zh/engines/zeta/state-storage-and-recovery.md`.
- **F1:** your diagnosis was right. It is fixed in source on this head and unverified at runtime because of N1. The log excerpt you asked for is exactly the N3 acceptance list.
# 5. Compatibility, deletions, load
- Unchanged from the last round: no wire, checkpoint, savepoint or `TaskExecutionState` format change; old members never write a marker and both consumers then take the `dev` `ERROR` path, so rolling upgrade is safe. With Issue 3 reverted there is no longer any change to shutdown semantics or shipped configuration.
- Deletion audit: the 34 deleted lines are the replaced `errorByPhysicalVertex` handling and inline `JobException` construction, `java` to `exec java`, and the two Kubernetes `command` lines. No connector, registration, config field or fallback was removed.
- Load: one `put` per intentional shutdown, one `get` plus one conditional `remove` per member removal, all on a TTL-bounded map. The per-vertex marker read on the master-failover restore path is unchanged and still not blocking.
# 6. Merge recommendation
**Blocking:**
1. N1: restore the `TimeUnit` import and get a complete CI run on the fixed head.
2. After that run, the N3 evidence must be present in the `engine-v2-it` log. Green status alone is not accepted for this PR; that is what let the inert implementation through last time.
**Recommended in the same PR:** N2 (bounded async marker calls). **Optional:** N4.
**Conclusion: implementation deviates from the plan.** At source level the change now matches what the PR describes, and both P0s are addressed the right way. The deviation is the Validation claim: the head does not compile, so none of the tests it cites have run. Posted as a comment because GitHub does not allow a formal review state on one's own PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Zeta] Log node-offline task failures as warning [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11856:
URL: https://github.com/apache/seatunnel/pull/11856#issuecomment-5846260494
Hi @SEZ9, thanks for the clear breakdown of what's still open. Head is still `dc2c64103c`, unchanged since my last comment, so nothing below required re-checking the source — just closing the gaps you flagged.
**F1**
- Code comment: agreed, and it isn't landed yet. I'll add it to `markLocalGracefulMemberRemoval()` in `SeaTunnelServer.java` in the next push, stating that correctness depends on `LifecycleServiceImpl.shutdown()` firing `SHUTTING_DOWN` before distributed-object proxies stop being served, and naming the Hazelcast version (5.1) this was verified against.
- Per-method result: still don't have it, and I won't claim more than I have. All I've confirmed is the class-level `Tests run: 10, Failures: 0, Errors: 0, Skipped: 2` for `ClusterFaultToleranceIT` from run `35240527669`. The next push (which also carries the fix below) will get a fresh CI run — I'll pull the per-method surefire line for `testGracefulShutdownPublishesMemberRemovalMarker` from that run and confirm it isn't one of the two skips before calling F1 closed.
**F2** — restating in full since it cut off again:
`CoordinatorService.java:2185` does a non-destructive `IMap.get`. The value-conditional `remove` at `:2207` only runs after the propagation loop finishes (`:2222-2238`), and only when `canClearGracefulMemberRemovalMarker` allows it (`:2239-2245`). So for the event that reads the marker, it is not removed before that event's own task-failure processing completes — that's (a).
For (b), master failover between read and clear: the new master doesn't depend on the old master's in-flight clear at all. It re-derives classification independently during restore, through `PhysicalVertex.checkTaskGroupIsExecuting`, which reads the same marker without clearing it (`PhysicalVertex.java:278-288`). So a failover mid-processing does not lose the graceful classification by itself.
What it does expose is a real bug I found on this same pass: `restoringRunningJobsFromMasterSwitch` is only reset inside `initCoordinatorService()`, never after a restore actually completes. So after the first master-switch restore with active jobs, `canClearGracefulMemberRemovalMarker` never returns `true` again for the rest of that master's term — the explicit post-processing clear becomes dead code, and every later removal falls back to waiting out the 5-minute TTL instead of being cleared immediately. That's the one thing I'm treating as blocking right now, and it's going into the same next push as the F1 comment, with a regression test that drives two failovers and asserts the clear fires both times.
**F3–F8** — fair to ask again, and I owe you F4 specifically: I skipped it in my last comment without flagging that I was skipping it. That one wasn't a rendering issue on your end, just an omission on mine — apologies for adding to the confusion given the genuine cut-offs elsewhere in this thread. Full status, nothing changed on any of these since `dc2c64103c`:
- F3/F6 (JobException vs. String payload): resolved. `buildMemberRemovedFailureState` wraps the message in `JobException` on both branches (`CoordinatorService.java:2144-2149`), pinned by `shouldKeepThrowablePayloadForMemberRemovedFailureState`.
- F4 (eviction / clock dependency): resolved. Eviction is a Hazelcast-side per-entry TTL on the `put` itself (`SeaTunnelServer.java:309-313`, `Constant.GRACEFUL_MEMBER_REMOVAL_MARK_TTL_MILLIS` = 5 minutes) — the entry expires on its own whether or not anything ever reads it. `isGracefulMemberRemovalMarkerValid` (`CoordinatorService.java:2157-2161`) is a separate, secondary check on top of that: it compares the departing member's timestamp against the coordinator's clock and rejects anything more than 5 minutes skewed. That part does depend on clocks roughly agreeing, but it fails in the safe direction — a marker that fails this check just falls back to `ERROR`, it never gets treated as graceful.
- F5/F8 (message-pattern matching / duplicated template): resolved. No `Pattern` or `.matches(` anywhere in `PhysicalVertex.java`; classification is a typed boolean threaded through from the marker lookup, and the offline message string has exactly one producer, `buildMemberRemovedOfflineMessage` (`CoordinatorService.java:2133-2137`).
- F7 (docs): resolved. Both `docs/en/engines/zeta/state-storage-and-recovery.md` and the `docs/zh` counterpart carry the marker/TTL/WARN-vs-ERROR section.
So, directly answering "next push vs. defer": everything above except the `restoringRunningJobsFromMasterSwitch` reset bug and the version-dependency comment is already landed, and I'm not touching it again. Those two go into the next commit together, along with the regression test and the per-method CI evidence for F1. I'll follow up on this thread once that's pushed rather than asking you to re-derive status from a comment again.
One more thing on the rendering pattern itself, since this is now the third time a comment in this thread has shown up cut off on your side (the interleaving explanation in August, the F4 explanation on the 20th, and my last comment on the 24th), while the stored body has checked out complete via the API every time I've verified it: I don't have visibility into what's causing that on your end, so I can't fix it directly, but I'll keep future updates shorter and split across separate comments where practical, since that seems like the more robust workaround until the underlying cause is understood.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Zeta] Log node-offline task failures as warning [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11856:
URL: https://github.com/apache/seatunnel/pull/11856#issuecomment-5379365917
Since this is my own PR, GitHub won't let me submit a formal review state on it, so this is posted as a plain comment again. This is round 7 on this PR (my sixth pass plus @SEZ9's independent review). The code itself is unchanged since my last comment (head is still `f953140ec`), so I'm not re-deriving the whole design from scratch again — I independently re-verified the current source against every load-bearing claim from round 6 and @SEZ9's review (all confirmed accurate), and I re-checked CI live rather than reusing round 6's "still queued" status. That live check changes the conclusion: **CI has since completed and the current head does not compile.** This is a hard blocker that supersedes my own round-6 "Ready to merge after fixes."
# What Problem Does This PR Solve?
When a worker leaves the Hazelcast cluster during a routine, operator-initiated scale-down, `CoordinatorService` fails its deployed task groups and `PhysicalVertex.stateProcess()` always logged that at `ERROR`, making expected scale-down noise indistinguishable from a real engine fault. After six rounds of iteration (fixing a message-construction bug that made the original classification a no-op, narrowing "graceful" from "any member removal" to a self-reported per-address marker, gating the marker on Hazelcast's `terminate` flag, and — following @SEZ9's finding — relocating the marker publication to the correct pre-shutdown SPI hook), the mechanism now publishes a best-effort "I'm leaving on purpose" marker into a new `engine_gracefulMemberRemoval` IMap before a graceful shutdown, consumes it when classifying the resulting task failure, and downgrades only the exact node-offline message to `WARN` when the marker proves the departure was graceful. Any unproven/abrupt departure
(crash, OOM, forced termination, partition) still logs `ERROR`, unchanged from today.
# 1. Code Change Review
## 1.1 Core Logic Analysis
I independently re-verified the current diff against the tree directly (not by re-reading the prior rounds' write-ups) and confirm every load-bearing claim from round 6 and @SEZ9's review holds:
- `SeaTunnelServer.java`: `SeaTunnelServer` now additionally implements `GracefulShutdownAwareService`; `onShutdown(long, TimeUnit)` calls `markLocalGracefulMemberRemoval()`, and the marker-write call is removed entirely from `ManagedService.shutdown(boolean terminate)` (confirmed by reading the full `shutdown()` body — every remaining line there operates on separately-mocked, null-guarded fields, none of them `nodeEngine`). `init()` clears this node's own stale marker on startup.
- `CoordinatorService.java`: adds `gracefulMemberRemovalIMap`, wired in `initCoordinatorService()`; `buildMemberRemovedOfflineMessage`/`buildMemberRemovedFailureState`/`isGracefulMemberRemovalMarkerValid`/`consumeGracefulMemberRemovalMarker` are exactly as described — the graceful branch builds `TaskExecutionState` via the `String` constructor (plain message, regex-matchable), the non-graceful branch still wraps in `new JobException(...)` (full stack trace, never matches, `ERROR` unchanged). `failedTaskOnMemberRemoved` calls `consumeGracefulMemberRemovalMarker` once, up front, and threads the resulting boolean through both `makeTasksFailed` call sites.
- `PhysicalVertex.java`: `DEPLOYED_NODE_OFFLINE_ERROR_PATTERN` and `isDeployedNodeOfflineFailure()` are exactly as described; `stateProcess()`'s `FAILED` case now branches `log.warn` vs `log.error` on that predicate. Confirmed the message-template is genuinely defined in two disconnected places with no shared constant (`CoordinatorService.java`'s `%s`/`%s` format string vs. `PhysicalVertex.java`'s hand-written regex) — @SEZ9's Issue 8 is real and I could find no test that feeds one into the other.
- `Constant.java`: adds `IMAP_GRACEFUL_MEMBER_REMOVAL` and a 5-minute `GRACEFUL_MEMBER_REMOVAL_MARK_TTL_MILLIS`, both with adequate Javadoc this round (closing part of round-6's Issue 3 from earlier rounds).
**Runtime path** (unchanged from round 6's trace, re-confirmed against current source):
```text
Graceful path:
operator-initiated shutdown -> Hazelcast Node.shutdown(false)
-> callGracefulShutdownAwareServices() -> SeaTunnelServer.onShutdown() [SeaTunnelServer.java:233-237]
markLocalGracefulMemberRemoval() -> gracefulMemberRemovalIMap.put(thisAddress, now)
-> (network) coordinator observes memberRemoved
-> CoordinatorService.failedTaskOnMemberRemoved(event) [CoordinatorService.java:2039]
gracefulMemberRemoval = consumeGracefulMemberRemovalMarker(lostAddress) [:2008-2013]
-> makeTasksFailed(..., gracefulMemberRemoval)
-> buildMemberRemovedFailureState(...): plain-String TaskExecutionState when graceful
-> PhysicalVertex.stateProcess() case FAILED: isDeployedNodeOfflineFailure(errorMsg) matches -> log.warn
Abrupt path (kill -9 / OOM / partition / forced terminate):
no marker ever written for that address -> consumeGracefulMemberRemovalMarker returns false
-> buildMemberRemovedFailureState(..., false) -> JobException(offlineMessage) -> stack trace
-> isDeployedNodeOfflineFailure() never matches a multi-line stack trace -> log.error (unchanged from dev)
```
**What I found that neither round 6 nor @SEZ9 caught: the current head does not compile.** I pulled the live CI run for the current head SHA (`f953140ecf6ee1e96544679c0ed28d91e51d94b0`, `DanielLeens/seatunnel` fork run `32484015488`, completed 2026-08-22, `conclusion: failure`) rather than trusting round 6's "still queued at review time" note. All four `unit-test` matrix legs (JDK 8/11 x ubuntu/windows) fail, and I pulled the raw job log to find the actual cause rather than assuming it was another Windows/CDC-style environmental flake:
```
[ERROR] COMPILATION ERROR :
[ERROR] .../seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/SeaTunnelServerShutdownTest.java:[55,16] error: no suitable method found for thenReturn(IMap<Address,Long>)
[ERROR] method OngoingStubbing.thenReturn(IMap<Object,Object>) is not applicable
[ERROR] (argument mismatch; IMap<Address,Long> cannot be converted to IMap<Object,Object>)
```
This is a `mvn test-compile` failure in `seatunnel-engine-server`, at:
```java
IMap<Address, Long> gracefulMemberRemovalIMap = mock(IMap.class); // raw-typed mock, unchecked cast
...
when(hazelcastInstance.getMap(Constant.IMAP_GRACEFUL_MEMBER_REMOVAL))
.thenReturn(gracefulMemberRemovalIMap); // javac can't reconcile the generics here
```
`mock(IMap.class)` produces a raw `IMap`; assigning it to the parameterized local variable is an unchecked conversion, and when it's then passed into `.thenReturn(...)` on the `OngoingStubbing<IMap<Object,Object>>` inferred from `HazelcastInstance.getMap(String)`'s unbound type parameters, javac 8/11 both fail to unify `IMap<Address,Long>` with `IMap<Object,Object>`. I confirmed this same single root cause cascades into the entire CI matrix, not just `unit-test`: I independently pulled the log for `all-connectors-it-1 (8, ubuntu-latest)` — a job with zero topical relation to this PR — and it shows the identical `COMPILATION ERROR` at the identical line, because that job's Maven invocation also builds `seatunnel-engine-server` with `-am` in the shared reactor. That is why essentially the entire 60+-job CI matrix is red on this head, not because of 60 unrelated flakes.
**Key findings:**
- The design and runtime-path logic (the actual subject of six rounds of review) remain sound and unchanged — the compile break is confined to one test file and does not reflect a defect in the production code path.
- This is a real, reproducible compile failure on this exact head, not a flake — it is 100% deterministic (a `javac` generics-inference limitation, not a race or environment issue), and it fully explains the "mass CI failure" pattern rather than requiring 60 separate unrelated-flake diagnoses.
- The fix is small and well-understood: either give `getMap` an explicit type witness (`hazelcastInstance.<Address, Long>getMap(...)`), or stub via `doReturn(gracefulMemberRemovalIMap).when(hazelcastInstance).getMap(...)` (which sidesteps `OngoingStubbing`'s generic-return-type inference entirely and is the more common idiom elsewhere in this codebase for exactly this class of Mockito+generics friction).
## 1.2 Compatibility Impact
Fully compatible. No config `Option`, public API, SPI contract break (implementing an additional Hazelcast-defined interface is additive), or checkpoint/wire format touched. Rolling-upgrade safety is preserved: an old-version member never implements `GracefulShutdownAwareService`, so its departure is never marked graceful and falls back to the pre-existing always-`ERROR` behavior identical to current `dev`.
## 1.3 Performance / Side-Effect Analysis
Negligible — one `IMap.put()` on graceful shutdown (now dispatched from Hazelcast's own bounded `graceful-shutdown` executor rather than inline in `ManagedService.shutdown()`), one `IMap.remove()` per `memberRemoved` event on the already-slow-path of task failure handling. No new locks, no hot-path allocation.
## 1.4 Error Handling and Logging
**Issue 1 — Current head does not compile; CI is red on a genuine compile error, not a flake**
- Location: `seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/SeaTunnelServerShutdownTest.java:55`
- Problem: `mock(IMap.class)` (raw type) assigned to `IMap<Address, Long>` and then passed to `.thenReturn(...)` fails javac's generic-return-type inference against `HazelcastInstance.getMap(String)`'s unbound `<K, V>` parameters, on both JDK 8 and JDK 11.
- Potential risk: this is not cosmetic — it blocks every CI job in the matrix (confirmed identical failure in an unrelated `all-connectors-it-1` job via the shared `-am` reactor build), and it must be treated as a release-blocking correctness gate regardless of how sound the surrounding design is.
- Best improvement: `Mockito.doReturn(gracefulMemberRemovalIMap).when(hazelcastInstance).getMap(Constant.IMAP_GRACEFUL_MEMBER_REMOVAL);` (avoids `OngoingStubbing`'s generic return-type inference entirely), or add an explicit type witness: `hazelcastInstance.<Address, Long>getMap(...)`.
- Severity: Critical
- Raised by another reviewer: No — CI was still queued at the time of both my round-6 review and @SEZ9's review; neither of us could have caught this from source reading alone since the code compiles under simple visual inspection (the type mismatch is a javac inference artifact, not a type error a human eye reliably catches). This is a genuine miss on my part in round 6 for asserting "no blockers remaining... contingent on CI" without following up once CI actually completed. I'm not going to let that stand uncorrected.
**Issue 2 (carried over from round 6, unaddressed) — `onShutdown()`'s comment misstates the mechanism it relies on**
- Location: `SeaTunnelServer.java:233-235`
- The comment says "before the node becomes PASSIVE"; per Hazelcast 5.1's `Node#shutdown(boolean)`, `NodeState` is set to `PASSIVE` unconditionally before `callGracefulShutdownAwareServices()` runs, so the node is already `PASSIVE` at that point — the actual reason the placement is safe is that it runs before `NodeEngineImpl.shutdown()` tears down `proxyService`/`operationService` (which back `getMap()`). I verified this against the actual Hazelcast 5.1 source in round 6 and it's still uncorrected in the current diff.
- Risk: low on its own; a future maintainer reasoning from the wrong premise could relocate this call to somewhere that again races proxy-service teardown.
- Best improvement: reword to describe the real mechanism (service-shutdown ordering, not `NodeState`).
- Severity: Low
- Raised by another reviewer: Partially (@SEZ9 raised the underlying placement concern; the comment-accuracy point is mine).
**Issue 3 (carried over, unaddressed) — marker consumption is destructive with no Hazelcast-native TTL**
- Location: `CoordinatorService.java` (`consumeGracefulMemberRemovalMarker`, `remove()` before any task group is failed), `SeaTunnelServer.java` (`put()` with no TTL overload).
- A master failover between `remove()` and the end of the propagation loop, or any departure whose marker is never read back, leaves an unbounded, un-evicted entry.
- Best improvement: use the TTL-bearing `put(key, value, ttl, unit)` overload; consider non-destructive read until the propagation loop completes.
- Severity: Medium
- Raised by another reviewer: Yes (@SEZ9, Issues 2 and 4).
**Issue 4 (carried over, unaddressed) — graceful branch swaps the `TaskExecutionState` payload from `JobException` to a bare `String`**
- Location: `CoordinatorService.buildMemberRemovedFailureState`.
- Downstream consumers of `throwableMsg` (job history, REST error fields) see a differently-shaped payload depending on which branch fired.
- Best improvement: keep `JobException` on both branches; carry the classification as an explicit flag instead of a payload-shape difference.
- Severity: Medium
- Raised by another reviewer: Yes (@SEZ9, Issues 3 and 6).
**Issue 5 (carried over, unaddressed) — offline-message template duplicated with no shared constant or round-trip test**
- Location: `CoordinatorService.buildMemberRemovedOfflineMessage` (format string) vs. `PhysicalVertex.DEPLOYED_NODE_OFFLINE_ERROR_PATTERN` (regex).
- I re-confirmed directly against the current tests: `PhysicalVertexTest` uses a hand-written literal, `CoordinatorServiceMemberRemovedTest` never feeds `buildMemberRemovedOfflineMessage`'s actual output into `isDeployedNodeOfflineFailure`. No test today would catch drift between the two.
- Best improvement: derive both from one shared constant, or add a round-trip test.
- Severity: Medium
- Raised by another reviewer: Yes (@SEZ9, Issue 8). This is the cheapest, highest-leverage of the non-blocking follow-ups.
**Issue 6 (carried over, unaddressed) — classification by free-text regex crosses a trust boundary**
- Location: `PhysicalVertex.isDeployedNodeOfflineFailure`, fed by `errorByPhysicalVertex`, which can also be populated from worker-reported/connector-originated messages.
- A connector exception whose message happens to match the exact template would be silently downgraded to `WARN`.
- Best improvement: carry the classification as a typed flag on `TaskExecutionState`, not message text.
- Severity: Medium
- Raised by another reviewer: Yes (@SEZ9, Issue 5).
**Issue 7 (carried over, unaddressed) — no docs/en / docs/zh update for the new IMap / WARN-vs-ERROR classification**
- Severity: Low
- Raised by another reviewer: Yes (@SEZ9, Issue 7).
# 2. Code Quality Assessment
## 2.1 Coding Standards
ASF headers present, Javadoc present on all new non-trivial fields/methods (a real improvement over round 2's gap), no wildcard imports, no `System.out.println`. The one standards issue is Issue 1 above — this specific test does not compile.
## 2.2 Test Coverage and Test Stability
The classification logic (`isDeployedNodeOfflineFailure`), the message-construction branching (`buildMemberRemovedFailureState`), and the marker-write/no-write branching in `SeaTunnelServerShutdownTest` are all exercised as plain, synchronous, mock-based unit tests — no `Thread.sleep`, no shared static state, no timing dependence, no containers/network I/O. Absent the compile error, this would rate **Stable**. Because of Issue 1, **the actual current rating must be: the test suite does not build, and therefore provides zero coverage on this head** — a compile failure is a stronger negative signal than any flaky-test pattern, since it means none of these otherwise well-designed tests actually ran. I'm not going to soften this by rating the *design* of the tests separately from the fact that they don't compile; the mandatory rating for this PR as it stands is **High risk**, purely on the compile break, with a clear and narrow one-line fix.
The remaining, pre-existing coverage gap (unchanged from round 6): no integration-level test exercises the real distributed `IMap` write/read round trip through an actual multi-node Hazelcast cluster — both my round-2 review and @SEZ9 flagged this; still open, non-blocking.
## 2.3 Documentation Updates
Not applicable for this specific commit's production code; see Issue 7 for the pre-existing, PR-wide documentation gap (operator-visible WARN/ERROR classification change, new IMap).
# 3. Architectural Soundness
## 3.1 Elegance of the Solution (Precise fix / Temporary workaround / Long-term solution)
The design itself is a **precise fix** for the stated problem, verified independently across six rounds including against actual upstream Hazelcast 5.1 source for the SPI-ordering question @SEZ9 raised. It remains fundamentally a message-text classification at the `PhysicalVertex` boundary (Issues 4-6), which is a reasonable minimal-diff tradeoff for this PR's scope, not a design defect. The one thing keeping this out of "long-term solution" territory today is purely the build break in Issue 1, which is mechanical, not conceptual.
## 3.2 Maintainability
Good and improving round over round, aside from the still-open comment-accuracy nit (Issue 2) and the message-duplication risk (Issue 5) that a future edit could silently regress.
## 3.3 Extensibility
Unchanged from round 6 — the self-reported marker pattern generalizes to future "was this expected?" classifications if needed.
## 3.4 Historical-Version Compatibility
No impact — no serialized state, checkpoint, or wire format touched; old, marker-unaware members simply never get a marker, unchanged always-`ERROR` fallback.
# 4. Issue Summary
| # | Issue | Location | Severity | Raised by another reviewer |
|---|---|---|---|---|
| 1 | Current head fails `test-compile` in `seatunnel-engine-server` (Mockito/javac generics inference), cascading a red CI matrix across ~60 unrelated jobs via the shared `-am` reactor build | SeaTunnelServerShutdownTest.java:55 | Critical | No |
| 2 | `onShutdown()`'s comment misstates the real mechanism (claims "before PASSIVE"; actually before proxy/operation-service teardown, while already PASSIVE) | SeaTunnelServer.java:233-235 | Low | Partially (@SEZ9) |
| 3 | Marker consumption destructive, no Hazelcast-native TTL; failover/never-consumed entries can leak or misclassify | CoordinatorService.java, SeaTunnelServer.java | Medium | Yes (@SEZ9) |
| 4 | Graceful branch swaps `JobException` payload for a bare `String`, inconsistent with the sibling branch | CoordinatorService.java | Medium | Yes (@SEZ9) |
| 5 | Offline-message template duplicated (format string vs. regex), no shared constant or round-trip test | CoordinatorService.java, PhysicalVertex.java | Medium | Yes (@SEZ9) |
| 6 | Classification by free-text regex match crosses a trust boundary | PhysicalVertex.java | Medium | Yes (@SEZ9) |
| 7 | No docs/en / docs/zh update for the new IMap / WARN-vs-ERROR classification | N/A | Low | Yes (@SEZ9) |
# 5. Merge Recommendation
### Conclusion: Not recommended for merge
**Blockers — must be fixed:**
1. Issue 1 (Critical) — fix the Mockito/generics compile break in `SeaTunnelServerShutdownTest.java:55` (`doReturn(...).when(...)` or an explicit type witness on `getMap`), then get a fully green CI run on the resulting head before merge. This is the sole reason I'm not endorsing my own round-6 "Ready to merge after fixes" today — that conclusion was reached while CI was still queued, and the completed run tells a different story than I assumed it would.
**Recommended fixes — non-blocking, all carried over and still valid:**
- Issue 5 — add a round-trip test (or shared constant) tying the message format string to the regex; cheapest, highest-leverage of the remaining items.
- Issue 3 — native-TTL `IMap.put` overload, non-destructive-until-processed marker consumption.
- Issue 4 / Issue 6 — carry the graceful/non-graceful classification as an explicit typed flag rather than a payload-shape or free-text difference; would close both at once.
- Issue 2 — correct the `onShutdown()` comment.
- Issue 7 — a short docs note on the WARN/ERROR distinction for operators who alert on Zeta engine logs.
**Overall assessment:** the design work across six rounds — including a second independent reviewer's real catch and a verified-correct fix for it — has been genuinely solid, and I stand by everything I said about the runtime logic in round 6: it fails safe (every ambiguous case defaults to the pre-existing `ERROR` behavior), and the mechanism itself is sound. But "ready to merge after fixes, contingent on CI" was not actually satisfied by this head — CI finished, and it's red for a real, deterministic reason. I should have circled back and confirmed the run before writing that conclusion instead of asking a maintainer to check it independently; I'm correcting that now rather than letting a stale "no blockers" stand on the PR. Once Issue 1 is fixed and CI is actually green on the resulting commit, I'd be comfortable with a maintainer merge decision — the remaining Medium/Low items are legitimate hardening work, not correctness gates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Zeta] Log node-offline task failures as warning [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11856:
URL: https://github.com/apache/seatunnel/pull/11856#issuecomment-5724160863
Thanks for the follow-up. The latest commit `dc2c64103c` (restoring the `TimeUnit` import in `SeaTunnelServerShutdownTest.java`) is a one-line, test-only change that fixes the compile break from `f3308d8168`; no objection to that part.
On the earlier findings, here is where I stand:
- **F1 (marker published from `ManagedService.shutdown()` while the node is PASSIVE):** the thread now describes the marker being written from a synchronous `LifecycleListener.stateChanged(SHUTTING_DOWN)` callback while the node is still `ACTIVE`, which is the right approach in principle. Could you point me to where the listener is registered and where the IMap write happens, and confirm there is a test that exercises the real Hazelcast lifecycle path rather than only a mocked map call?
- **F4 (no eviction on `engine_gracefulMemberRemoval`):** the thread mentions a "TTL-bounded marker". Is that a Hazelcast-side TTL on the map entry, or still a manual timestamp check that depends on cross-node clock agreement? If the former, F4 is resolved; if the latter, it remains open.
- **F2 (marker consumed before failure processing / master failover):** the thread says `PhysicalVertex` classifies directly for the master-failover restore path. Is the marker still destructively removed before task-failure processing completes, and how does a failover mid-processing keep the graceful classification?
- **F3 / F6 (`JobException` replaced by a plain String in `TaskExecutionState`):** no change described. Are these still as originally reported? If so, I'd like the graceful path to keep the `JobException` (or an equivalent throwable) so the persisted/exposed error payload and job history/REST fields are unchanged.
- **F5 / F8 (free-text message matching, template duplicated between `CoordinatorService` and `PhysicalVertex`):** if `PhysicalVertex` now classifies using the marker rather than matching the offline message, please confirm the regex/template duplication has been removed. Otherwise both remain open.
- **F7 (docs):** I still don't see documentation for the new IMap and the WARN downgrade behavior under `docs/`. Please add it or tell me where it lives.
A short per-finding status would be great, and I'll do a final pass on the current head once I have it.
<!-- streview-comment:1125 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Zeta] Log node-offline task failures as warning [seatunnel]
Posted by "SEZ9 (via GitHub) via commits" <co...@seatunnel.apache.org>.
SEZ9 commented on PR #11856:
URL: https://github.com/apache/seatunnel/pull/11856#issuecomment-6870898385
Quick status update on this PR.
The head `b32e5360d881cc9cdf889a9a44370767129babf3` is now reported as having merge conflicts with `dev` (`mergeable_state=dirty`), so the earlier note in review 5821101205 that it merged cleanly no longer holds. Please rebase onto the latest `dev` and resolve the conflicts, keeping both the intended graceful-shutdown change and the upstream changes intact. Since this will produce a new head, I'll re-check the resolved version in full rather than diffing against the old one.
Independently of the conflict, the points from the previous review are still open and nothing in the thread so far addresses them:
- **PR11856-F1 (HIGH):** publishing the graceful-shutdown marker from `ManagedService.shutdown()` in `SeaTunnelServer` happens when the Hazelcast node is already PASSIVE, so the IMap write can be rejected and the WARN downgrade silently no-ops in real graceful shutdowns. This one is the key fix — please move the marker publish to a point where the node can still write, and ideally add a test that exercises a real graceful shutdown path.
- **PR11856-F2 / F4:** the marker is destructively consumed before task-failure processing (losing the classification on master failover mid-processing) and entries in `engine_gracefulMemberRemoval` have no eviction, relying on a manual TTL check across node clocks. Please consider a non-destructive read plus a proper IMap TTL/eviction config.
- **PR11856-F3 / F6:** the graceful path swaps the `JobException` in `TaskExecutionState` for a plain String, which changes the persisted/REST error payload and contradicts the "state handling unchanged" claim. Please keep the throwable and only adjust the log level.
- **PR11856-F5 / F8:** classification by matching the free-text message in `PhysicalVertex` (with the template duplicated as a format string in `CoordinatorService` and a regex in `PhysicalVertex`) is fragile and spoofable. A typed signal (dedicated exception type or flag) would remove both the drift risk and the mimicry concern.
- **PR11856-F7:** please add a short docs/ section describing the new marker IMap and the WARN downgrade behavior.
Once the rebase is pushed, a short note on which of the above are addressed in the new head would help speed up the re-review. Thanks for working through this!
<!-- streview-comment:2339 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Zeta] Log node-offline task failures as warning [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11856:
URL: https://github.com/apache/seatunnel/pull/11856#issuecomment-5421773819
Thanks for flagging that, @SEZ9 — I checked the raw comment body via the API and the full text is actually there on GitHub's side (18,939 characters, nothing truncated in storage), so this looks like a rendering/fetch issue on your end rather than something lost when I posted it. Re-pasting the two missing pieces directly so you don't have to chase it further.
## The race (Issue 1, High)
**Root cause:** `PhysicalVertex.updateStateByExecutionService()` writes two independent atomics as an unsynchronized pair, with mismatched CAS discipline:
```java
// PhysicalVertex.java:546-556
public void updateStateByExecutionService(
TaskExecutionState taskExecutionState, boolean gracefulMemberRemovalFailure) {
...
errorByPhysicalVertex.compareAndSet(null, taskExecutionState.getThrowableMsg()); // first-write-wins
gracefulMemberRemovalFailureByPhysicalVertex.set(gracefulMemberRemovalFailure); // last-write-wins
updateTaskState(taskExecutionState.getExecutionState());
}
```
`errorByPhysicalVertex` is `compareAndSet(null, ...)` — first caller wins the message, by pre-existing design. `gracefulMemberRemovalFailureByPhysicalVertex` (new in this PR) is a plain `.set(...)` — last caller wins the flag. They are not updated as one unit, and the method has no `synchronized` guard (unlike `updateTaskState()`/`stateProcess()`, which are both `synchronized`).
**Concrete interleaving:**
```
Thread A (Hazelcast operation-executor thread — a real task failure reported over RPC):
NotifyTaskStatusOperation
-> JobMaster.updateTaskExecutionState(taskExecutionState) [JobMaster.java:1253/1267]
-> physicalVertex.updateStateByExecutionService(taskExecutionState) // 1-arg overload
-> updateStateByExecutionService(taskExecutionState, false) [PhysicalVertex.java:539]
errorByPhysicalVertex.compareAndSet(null, "<real exception message>") // succeeds
gracefulMemberRemovalFailureByPhysicalVertex.set(false)
Thread B (Hazelcast membership-event thread — same vertex, classified node-offline because
this same worker is also mid-graceful-shutdown around the same time):
CoordinatorService.failedTaskOnMemberRemoved(event) [CoordinatorService.java:2030]
-> makeTasksFailed(..., gracefulMemberRemoval=true) [CoordinatorService.java:2057]
(guard: executionState == RUNNING — still true if Thread A hasn't transitioned yet)
-> physicalVertex.updateStateByExecutionService(
buildMemberRemovedFailureState(...), true) [CoordinatorService.java:2071]
errorByPhysicalVertex.compareAndSet(null, "...") // FAILS silently, A already won
gracefulMemberRemovalFailureByPhysicalVertex.set(true) // overwrites A's `false`
```
If Thread B's `.set(true)` lands after Thread A's `.set(false)` — entirely possible, since `NotifyTaskStatusOperation` (operation-executor pool) and `MembershipAwareService.memberRemoved` (cluster/membership-event thread) are different Hazelcast thread pools with no ordering guarantee between them — then whichever thread wins the race into `synchronized updateTaskState()` and actually flips `RUNNING -> FAILED` calls `stateProcess()`, which reads:
- `errorByPhysicalVertex.get()` = Thread A's real exception message (correct)
- `gracefulMemberRemovalFailureByPhysicalVertex.get()` = `true`, from Thread B
Net effect: a genuine task failure, carrying its own genuine error message, gets logged at `log.warn` instead of `log.error` (`PhysicalVertex.java:632-641`). The message text is still the real exception, so a human reading the raw line isn't misled about *what* failed — but the severity, which is exactly the signal this PR exists to make trustworthy, is wrong. The losing thread hits the pre-existing `current.isEndState()` guard and returns silently, so there's no double-log to notice the discrepancy from.
**Fix options:**
- Option A (more robust): collapse both fields into one `AtomicReference<FailureClassification>` (an immutable `{message, gracefulFlag}` holder) updated via a single `compareAndSet(null, ...)`, so message and flag always travel together under first-write-wins.
- Option B (minimal diff): only write the flag when the paired message-write actually wins:
```java
if (errorByPhysicalVertex.compareAndSet(null, taskExecutionState.getThrowableMsg())) {
gracefulMemberRemovalFailureByPhysicalVertex.set(gracefulMemberRemovalFailure);
}
```
One-line guard, restores "the flag belongs to whichever call's message actually stuck."
I lean toward Option A since this is state future maintainers will likely extend, but Option B closes the actual bug on its own.
## CI status at review time
`Build` was failing on fork run `32628391769` (head `06344bd0`). Pulled the job log directly: every module through `seatunnel-engine-server` (the module this PR touches) builds and unit-tests green. The only failing job is `rocketmq-connector-it (11, ubuntu-latest)`, one failed test — `RocketMqIT.testSourceRocketMqRestore`, `Expected 20 '_initial_' messages, got: 25` — preceded in the log by `RemotingSendRequestException: send request to <172.18.0.2:9876> failed` against the RocketMQ test container. That reads as a message-count/duplicate-delivery flake in an unrelated connector's E2E suite, not something a `seatunnel-engine`-only diff could cause. Still needs to go green (or be explicitly waived as known flake) before merge, per the usual bar.
Full text is also on the original review if the render issue clears up: https://github.com/apache/seatunnel/pull/11856#pullrequestreview-5009299593
Once the pairing fix (Option A or B) is in, I'll re-verify against the new head.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Zeta] Log node-offline task failures as warning [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11856:
URL: https://github.com/apache/seatunnel/pull/11856#issuecomment-5707770751
Thanks for the rebase. `fd60dfd596` is a pure `dev` merge on top of `e6e243744e` with content identical to the previously reviewed head, so the earlier findings still apply. As noted above, the "Ready to merge" conclusion posted at `e6e243744e` is retracted: green CI alone was not sufficient evidence, and the engine E2E logs show the WARN downgrade has not fired on the graceful-shutdown path.
**Blocking**
- **PR11856-F1 — marker published too late.** `SeaTunnelServer.onShutdown()` runs from the `GracefulShutdownAwareService` hook, after the node has already moved to `PASSIVE`, so the `engine_gracefulMemberRemoval` write is rejected and the coordinator never sees a marker. Please move the marker write to a point where the node is still `ACTIVE` (for example the synchronous `SHUTTING_DOWN` lifecycle event) and add an E2E assertion for the WARN-level offline log line so this cannot regress silently.
**Should be addressed before merge (medium)**
- **PR11856-F3 / PR11856-F6 — payload change on the graceful path.** `CoordinatorService` replaces the `JobException` throwable in `TaskExecutionState` with a plain string, so job history / REST error fields lose the stack trace. That contradicts the "failure state handling unchanged" statement in the PR body. Please keep the throwable and only narrow the log level.
- **PR11856-F2 / PR11856-F4 — marker lifecycle.** The marker is destructively consumed before task-failure processing (a master failover mid-processing loses the graceful classification), and the IMap has no eviction; the TTL is only a manual timestamp check that depends on cross-node clocks agreeing. Please consume after processing (or make it idempotent) and configure a real per-entry TTL on the map.
- **PR11856-F5 / PR11856-F8 — message-based classification.** The offline template exists as a format string in `CoordinatorService` and as a regex in `PhysicalVertex`; any free-text failure that happens to match gets downgraded, and the two definitions can drift independently. A typed marker (dedicated exception subclass or an explicit flag on the state) would address both.
- **PR11856-F7 — docs.** The new `engine_gracefulMemberRemoval` IMap and the WARN downgrade are cluster-visible behavior; please add a short note under `docs/`.
Once F1 is fixed, could you paste the engine E2E log excerpt showing the marker write succeeding and the offline failure logged at WARN? That is the evidence missing at `e6e243744e`, and it would be good to have in the thread before re-approving.
<!-- streview-comment:1113 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Zeta] Log node-offline task failures as warning [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11856:
URL: https://github.com/apache/seatunnel/pull/11856#issuecomment-5701028678
[REVIEW] Maintainer re-review of the current head `fd60dfd596` (a pure `dev` merge on top of `e6e243744e`; I verified the three-dot diff against `dev` is content-identical to the previously reviewed head, 31 files, +1148/-42).
**Verdict: not ready to merge. Two blocking findings, both confirmed from the real Hazelcast 5.1 source and from the engine E2E logs of the last green CI run.** This retracts the "Ready to merge" conclusion I posted on 2026-09-06 at `e6e243744e`. That conclusion rested on green CI plus source-level tracing of the `GracefulShutdownAwareService` hook. It never checked what the engine E2E logs actually recorded on the graceful-shutdown path. I did that this round, and the logs contradict the earlier conclusion: the WARN downgrade has never fired in CI, and the marker write fails on every graceful shutdown.
# What this PR is supposed to do
When a worker leaves the cluster during an intentional scale-down, `CoordinatorService.failedTaskOnMemberRemoved` fails its task groups and `PhysicalVertex.stateProcess()` logs that at `ERROR`. The PR has a node that is shutting down gracefully write a TTL-bounded marker into the new `engine_gracefulMemberRemoval` IMap, and the coordinator downgrades only a marker-proven departure to `WARN`. The PR body says the change "keeps the existing failure state handling unchanged and only narrows the log level for the known node-offline case".
# Runtime chain re-traced on this head
SeaTunnel side is the PR head; Hazelcast side is the vanilla 5.1 sources jar. The shaded `seatunnel-shade-hazelcast:5.1-3.0.0` jar that CI runs carries byte-identical `Node.class` and `Invocation.class` (md5 checked), so the vanilla source is authoritative here.
```text
Graceful shutdown (intended marker write)
HazelcastInstance.shutdown() / NodeShutdownHookThread with policy GRACEFUL Node.java:751-779
-> LifecycleServiceImpl.shutdown(terminate=false) LifecycleServiceImpl.java:92-105
fireLifecycleEvent(SHUTTING_DOWN) (synchronous, node still ACTIVE) :94, :66-72
Node.shutdown(false) Node.java:502
setShuttingDown(): shuttingDown=true, state=PASSIVE Node.java:633-635
callGracefulShutdownAwareServices() (graceful-shutdown executor) Node.java:520, 554-579
-> SeaTunnelServer.onShutdown() SeaTunnelServer.java:242
-> updateLocalGracefulMemberRemovalMarker(true) SeaTunnelServer.java:310-343
-> IMap.put(thisAddress, now, TTL) SeaTunnelServer.java:319-323
-> MapProxyImpl.put -> AbstractDistributedObject.toData()
-> getNodeEngine() -> lifecycleCheck() AbstractDistributedObject.java:104-115
-> NodeEngineImpl.isRunning() -> Node.isRunning() NodeEngineImpl.java:394, Node.java:646-648
== !shuttingDown.get() == false
=> HazelcastInstanceNotActiveException caught at SeaTunnelServer.java:335
shutdownServices() -> SeaTunnelServer.shutdown(false)
Coordinator classification
MembershipManager.sendMembershipEventNotifications -> membership-event executor MembershipManager.java:819-833
-> SeaTunnelServer.memberRemoved (catches SeaTunnelEngineException only) SeaTunnelServer.java:349-357
-> CoordinatorService.memberRemoved CoordinatorService.java:2249-2254
-> failedTaskOnMemberRemoved CoordinatorService.java:2197-2226
-> getGracefulMemberRemovalMarker -> IMap.get (no try/catch) CoordinatorService.java:2179-2184
-> makeTasksFailed(...) CoordinatorService.java:2228-2246
-> clearGracefulMemberRemovalMarker -> IMap.remove (no try/catch) CoordinatorService.java:2191-2195
Log level
PhysicalVertex.stateProcess case FAILED -> shouldLogFailureAsWarn PhysicalVertex.java:666-692, 728-730
```
# Findings
## Issue 1 (P0, blocking): the marker write is unreachable on the real graceful-shutdown path. The WARN downgrade never fires, and every graceful shutdown now logs a new WARN with a stack trace.
**Trigger condition:** any graceful shutdown of a SeaTunnel node (`HazelcastInstance.shutdown()`, or SIGTERM under the new `GRACEFUL` hook policy). This is the exact scenario the PR targets.
**Root cause (Hazelcast 5.1 source):**
- `Node.shutdown()` flips `shuttingDown` to `true` in `setShuttingDown()` (`Node.java:633-635`) before it calls `callGracefulShutdownAwareServices()` (`Node.java:520`).
- Every `IMap` proxy method goes through `AbstractDistributedObject.getNodeEngine()`, whose `lifecycleCheck()` throws `HazelcastInstanceNotActiveException` when `!engine.isRunning()` (`AbstractDistributedObject.java:104-115`). `NodeEngineImpl.isRunning()` delegates to `Node.isRunning()`, which is `!shuttingDown.get()` (`NodeEngineImpl.java:394`, `Node.java:646-648`).
- Therefore no IMap proxy operation can succeed from inside `GracefulShutdownAwareService.onShutdown()`, deterministically, regardless of node state, cluster state, or partition ownership. The earlier rounds' claim that this hook runs "while map and operation services are still active" was wrong: the operation service is still running, but the proxy layer rejects the call.
**Empirical evidence (fork run `33966670706`, head `e6e243744e`, whose diff is content-identical to this head):**
| counter in the engine E2E job log | `engine-v2-it (8)` | `engine-v2-it (11)` |
|---|---|---|
| `Failed to mark graceful member removal marker` (SeaTunnelServer WARN) | 153 | 152 |
| of which followed by `HazelcastInstanceNotActiveException: Hazelcast instance is not active!` | 139 (rest are interleaved log lines) | 139 |
| `Graceful shutdown failed for org.apache.seatunnel.engine.server.SeaTunnelServer` (Hazelcast WARN) | 153 | 152 |
| `end with state FAILED due to node offline` (the new WARN branch) | **0** | **0** |
| `end with state FAILED and Exception: ... deployed node(...) offline` (the pre-PR ERROR branch) | 122 | 124 |
Sample stack from the JDK 8 log (`ClusterFaultToleranceIT`-family tests call `node.shutdown()` on real multi-node clusters):
```text
WARN org.apache.seatunnel.engine.server.SeaTunnelServer - Failed to mark graceful member removal marker
com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.hazelcast.spi.impl.AbstractDistributedObject.throwNotActiveException(AbstractDistributedObject.java:115)
at com.hazelcast.spi.impl.AbstractDistributedObject.lifecycleCheck(AbstractDistributedObject.java:110)
at com.hazelcast.spi.impl.AbstractDistributedObject.getNodeEngine(AbstractDistributedObject.java:104)
at com.hazelcast.spi.impl.AbstractDistributedObject.toData(AbstractDistributedObject.java:78)
at com.hazelcast.map.impl.proxy.MapProxyImpl.put(MapProxyImpl.java:137)
at org.apache.seatunnel.engine.server.SeaTunnelServer.updateLocalGracefulMemberRemovalMarker(SeaTunnelServer.java:319)
at org.apache.seatunnel.engine.server.SeaTunnelServer.markLocalGracefulMemberRemoval(SeaTunnelServer.java:296)
at org.apache.seatunnel.engine.server.SeaTunnelServer.onShutdown(SeaTunnelServer.java:243)
at com.hazelcast.instance.impl.Node$2.run(Node.java:564)
WARN com.hazelcast.instance.impl.Node - [localhost]:5801 [...] Graceful shutdown failed for org.apache.seatunnel.engine.server.SeaTunnelServer@362203b
```
**Impact:** the feature is inert in production. Classification stays at `ERROR` exactly as on `dev`, and on top of that every graceful shutdown emits two new WARN lines, one with a full stack trace. For a PR whose purpose is to reduce misleading log noise, this is a net regression. Green CI proves nothing here: the failure is caught and logged at WARN, so no test fails.
**Why the tests did not catch it:** `SeaTunnelServerShutdownTest.shouldMarkGracefulMemberRemovalOnGracefulShutdown` mocks `IMap`, so by construction it cannot observe the proxy's lifecycle check. This is the second time on this PR that the marker was published from a hook where the write cannot succeed (SEZ9's original finding was `ManagedService.shutdown()`); a mock-only test cannot guard SPI placement.
**Fix direction (minimal):** publish the marker from `LifecycleListener.stateChanged(SHUTTING_DOWN)`. `LifecycleServiceImpl.shutdown()` fires that event synchronously at `LifecycleServiceImpl.java:94`, before `node.shutdown()` at `:101`, so `node.isRunning()` is still `true` and node state is `ACTIVE`; a plain `IMap.put` works there. `SeaTunnelServer` is already registered as a `LifecycleListener` at `SeaTunnelServer.java:156`, so this removes the `GracefulShutdownAwareService` implementation rather than adding code. Trade-off to state explicitly in the PR: `SHUTTING_DOWN` fires for both `shutdown()` and `terminate()`. Both are intentional in-process exits (an API call or the JVM shutdown hook); crashes, `kill -9`, OOM kills and network partitions never fire it, so the "intentional versus unproven" distinction the PR cares about is preserved. If forced `terminate()` must still be excluded, that needs a different signal, and whatever is chosen must be proven with a real-cluster tes
t (Issue 7).
## Issue 2 (P0, blocking): a new unguarded distributed `IMap.get` at the head of `failedTaskOnMemberRemoved` can abort task-failure propagation for an entire lost member.
**Trigger condition:** any member removal (this is the normal path for every worker loss, graceful or not) where `gracefulMemberRemovalIMap.get(lostAddress)` throws.
**Root cause:** `failedTaskOnMemberRemoved` (`CoordinatorService.java:2197-2226`) now calls `getGracefulMemberRemovalMarker` (`:2179-2184`), a synchronous distributed `IMap.get`, before the `runningJobMasterMap.forEach(...)` loop, with no exception handling. The exception propagates through `CoordinatorService.memberRemoved` (`:2249-2254`) to `SeaTunnelServer.memberRemoved` (`SeaTunnelServer.java:349-357`), which only catches `SeaTunnelEngineException`, and then to the Hazelcast membership-event executor (`MembershipManager.java:831`), where it is swallowed as an executor-level log. Result: no task group on the lost worker is marked `FAILED`, the pipeline never restarts, and the job sits in `RUNNING` with tasks that no longer exist. There is no other detector for this: `checkTaskGroupIsExecuting` only runs on master-failover restore and on cancel.
**Why this is new:** on `dev`, everything before the per-vertex `updateTaskState` is in-memory (`getCurrentExecutionAddress` reads the job master's owned-slot map, `getExecutionState()` returns the `currExecutionState` field), and `updateTaskState` wraps its IMap access in `RetryUtils.retryWithException` plus a per-vertex `catch (Exception e)` (`PhysicalVertex.java:390-446`). So on `dev` a Hazelcast hiccup costs at most one vertex; on this head it costs every vertex of every job on the lost member.
**When `get` throws in practice:** right at member removal, Hazelcast is repartitioning the partitions the lost member owned. A `get` that lands on such a partition retries on `WrongTargetException`/`TargetNotMemberException` up to `hazelcast.invocation.max.retry.count`, which the shipped configs set to `20`, with the backoff in `Invocation.handleRetry` (`Invocation.java:706-712`): roughly 3.5 s of budget before the exception propagates. Slow owners hit `OperationTimeoutException` at the 60 s call timeout instead. At the load profile this engine is reviewed against (trillions of rows per day, tens of billions per job, 271 partitions repartitioning under checkpoint state pressure), a >3.5 s repartition at the moment a worker disappears is a plausible production event, not a corner case.
**Fix direction:** wrap the marker read (and the trailing `clearGracefulMemberRemovalMarker` at `:2222-2225`) in `try/catch` and treat any failure as "unproven" (`null` marker, `ERROR` path). `PhysicalVertex.getGracefulMemberRemovalMarker` (`PhysicalVertex.java:278-288`) already does exactly this on the restore path; the coordinator path must be at least as defensive, because it is the primary path.
## Issue 3 (Medium): `hazelcast.shutdownhook.policy: GRACEFUL` changes SIGTERM semantics for every shipped deployment, is unrelated to logging, and is unnecessary once Issue 1 is fixed correctly.
- Default policy is `TERMINATE` (`ClusterProperty.java:1533-1534`). The PR sets `GRACEFUL` in 3 shipped `config/*.yaml`, the engine-common default `hazelcast.yaml`, 2 Kubernetes conf files and 11 docs pages.
- `stop-seatunnel-cluster.sh` sends a plain `kill` (SIGTERM) at `:51` and `:58`. Under `GRACEFUL`, the JVM hook now runs `lifecycleService.shutdown()`, so `InternalPartitionServiceImpl.onShutdown` (`InternalPartitionServiceImpl.java:897-935`) blocks until the master has migrated this member's partitions away, bounded by `hazelcast.graceful.shutdown.max.wait` = 600 s (`ClusterProperty.java:627-628`, not overridden anywhere in the repo). Stopping a data member (master, or a hybrid MASTER_AND_WORKER node) with a large IMap footprint now keeps the JVM alive noticeably longer; on Kubernetes the pod can be SIGKILLed at `terminationGracePeriodSeconds` mid-migration.
- None of this is described in the PR body or in the added docs, which only say operators "must retain the GRACEFUL policy". A change to what SIGTERM does on every node needs its own justification, its own doc paragraph about the wait and its timeout, and a release-note entry.
- With the Issue 1 fix on `SHUTTING_DOWN`, the marker is written under the default `TERMINATE` policy as well (`terminate()` also goes through `LifecycleServiceImpl.shutdown(true)` and fires `SHUTTING_DOWN` at `:94`). I recommend dropping the 18 config/doc/Kubernetes edits from this PR, or splitting them into a dedicated PR with the rationale above.
## Issue 4 (Medium): the startup-side marker clear fails deterministically on lite-member workers and logs a WARN with a stack trace on every worker boot.
- `stateChanged(STARTING)` (`SeaTunnelServer.java:252-256`) issues `removeAsync` from inside `Node.start()` at `Node.java:460`, before `join()` at `:483`. Separated-cluster workers are lite members (`SeaTunnelServerStarter.java:101`), so pre-join the only known member is a lite member and `PartitionInvocation.newTargetNullException` throws `NoDataMemberInClusterException`, which extends `HazelcastException` and is not retryable (`NoDataMemberInClusterException.java:24`). The `.exceptionally` handler then logs a WARN with the full stack.
- Evidence: 35 `Failed to clear graceful member removal marker` in each engine E2E job (22 `NoDataMemberInClusterException`, 8 `WrongTargetException` after retries were exhausted on master nodes, 4 `HazelcastInstanceNotActiveException`) across 200 member starts.
- Functional impact is bounded (Hazelcast TTL plus the coordinator's post-classification clear), but this is another new per-boot WARN with a stack trace from a noise-reduction PR.
- Fix direction: fire the clear from a post-join lifecycle point. `LifecycleState.STARTED` is fired from `HazelcastInstanceFactory.constructHazelcastInstance` (`HazelcastInstanceFactory.java:241`) only after `Node.start()` has returned and the join completed, so the partition table has a data member to target. Keep it asynchronous and fail-safe as it is now.
## Issue 5 (Low): the cancel path now records an "offline" failure message where none existed before.
`checkTaskGroupIsExecuting` (`PhysicalVertex.java:228-271`) now installs `The taskGroup(...) deployed node(...) offline` into the failure slot whenever the worker is missing. It is called not only from the restore path but also from `noticeTaskExecutionServiceCancel` (`PhysicalVertex.java:457`), so a `CANCELED` task whose worker has already left now completes its `TaskExecutionState` with that message (`:658-664`) instead of `null`. Probably harmless, but it contradicts "failure state handling unchanged" and should be listed in the PR description as a deliberate change.
## Issue 6 (Low): Javadoc and docs assert the opposite of the observed ordering.
`SeaTunnelServer.onShutdown` Javadoc (`SeaTunnelServer.java:236-240`) and `docs/en|zh/engines/zeta/state-storage-and-recovery.md` state that the callback runs "while map and operation services are still active" and "writes each server member's address into this map before member removal". Issue 1 shows the IMap proxy rejects the write there. Both must be rewritten with the fix so the next reader is not misled the way this review thread was.
## Issue 7 (Low on its own, required with the fix): no real-cluster test covers the PR's central claim.
All three new test classes are Mockito-based. `ClusterFaultToleranceIT` and its siblings already shut real nodes down in multi-node clusters; adding an assertion there (marker present at the coordinator, or the `due to node offline` WARN line observed) would have failed on every revision of this PR so far. Please add that assertion together with the Issue 1 fix; without it the SPI placement cannot be considered verified.
# Compatibility and upgrade
- No wire, checkpoint, savepoint or `TaskExecutionState` format change; `FailureClassification` is in-process only. Rolling upgrade is safe: old nodes never write a marker, so both consumers take the pre-existing `ERROR` path.
- Operators with a custom `hazelcast.yaml` keep `TERMINATE` and silently keep the old behavior (documented, but see Issue 3 for why the policy change should not be in this PR at all).
- Kubernetes `command` change (`/bin/sh -c "<script>"` to direct script exec) and `exec java` in the foreground path are fine: the script already carried `#!/bin/bash` and was executed through its shebang before, so no new image dependency; the daemon (`-d`) path is untouched.
# Deletion audit
All 42 deleted lines are accounted for: the replaced `errorByPhysicalVertex` lines and the old inline `JobException` construction in `PhysicalVertex`/`CoordinatorService`, trailing-newline fixes in yaml/md files, the `java` to `exec java` line, and the two Kubernetes `command` lines. No connector, datasource, registration, config field or fallback was removed.
# Extreme-load assessment
Steady-state cost of the design is one `put` per graceful shutdown and one `get` plus one `remove` per member removal on the master, all against a TTL-bounded map: negligible. The restore path adds one synchronous `get` per vertex whose worker vanished (`PhysicalVertex.java:244-249`), serialized inside `initStateFuture`; for jobs with thousands of vertices on one lost worker this is thousands of sequential distributed reads during failover. Not blocking, but caching the marker per lost address in the restore path would be cheap. The blocking risk at scale is Issue 2, not throughput.
# Issue summary
| # | Issue | Severity | Location |
|---|---|---|---|
| 1 | Marker write from `GracefulShutdownAwareService.onShutdown()` always throws `HazelcastInstanceNotActiveException` (proxy lifecycle check after `shuttingDown=true`); WARN downgrade never fires; two new WARN lines per graceful shutdown. Confirmed by 0 WARN / 122 ERROR / 153 write failures in the last green engine E2E run | P0, blocking | `SeaTunnelServer.java:242,310-343`; `Node.java:502-520,633-635`; `AbstractDistributedObject.java:104-115` |
| 2 | Unguarded `IMap.get` before the failure-propagation loop in `failedTaskOnMemberRemoved`; a transient Hazelcast exception now skips failing every task on the lost member and can leave the job stuck | P0, blocking | `CoordinatorService.java:2179-2184,2197-2226`; `SeaTunnelServer.java:349-357` |
| 3 | `hazelcast.shutdownhook.policy: GRACEFUL` changes SIGTERM into a blocking graceful shutdown (up to 600 s) for every shipped deployment; undocumented; unnecessary once Issue 1 is fixed on `SHUTTING_DOWN` | Medium | 18 config/doc/k8s files; `stop-seatunnel-cluster.sh:51,58` |
| 4 | Startup-side clear at `STARTING` fails deterministically on lite-member workers (`NoDataMemberInClusterException`) and logs a stack trace on every worker boot | Medium | `SeaTunnelServer.java:252-256,325-332` |
| 5 | Cancel path now records an "offline" failure message on `CANCELED` states | Low | `PhysicalVertex.java:228-271,457,658-664` |
| 6 | Javadoc and docs describe an ordering that the source and logs disprove | Low | `SeaTunnelServer.java:236-240`; `docs/en|zh/engines/zeta/state-storage-and-recovery.md` |
| 7 | No real-cluster assertion for the marker/WARN path; all new tests are mock-based | Low, required with the fix | `ClusterFaultToleranceIT` or a dedicated IT |
# CI status on this head
- Fork `Build` run `35119618733` on `fd60dfd596` was still `queued` at review time; the apache-side checks on this commit are queued as well. No run-verification claim can be made for this head yet.
- The previous head `e6e243744e` was green (fork run `33966670706`, 83/83 non-skipped jobs). Green here does not validate this feature: the failing write is caught and logged, which is exactly why the E2E logs, not the job status, had to be read.
# Merge recommendation
**Blocking, must be fixed before any further review round:**
1. Issue 1: move the marker write to `stateChanged(SHUTTING_DOWN)` (or prove another placement with a real-cluster test), remove the `GracefulShutdownAwareService` implementation, and fix the Javadoc/docs (Issue 6).
2. Issue 2: make the coordinator's marker read and clear fail-safe, mirroring `PhysicalVertex.getGracefulMemberRemovalMarker`.
3. Issue 7: add a real multi-node assertion for the WARN classification so this cannot regress a third time.
**Recommended in the same PR:** drop or split out the `hazelcast.shutdownhook.policy` change (Issue 3); move the startup clear to `STARTED` (Issue 4); mention the cancel-path message change in the PR body (Issue 5).
**Conclusion: implementation deviates from the plan.** The PR body promises "only narrows the log level for the known node-offline case" while keeping failure handling unchanged. On the real path the log level never narrows, and the member-removed failure handling did change, in the wrong direction. I am keeping this review as a comment rather than a formal review state because GitHub does not allow a formal state on one's own PR. @SEZ9, your original Issue 1 was right in substance and is still open on this head: the hook moved, but the write still cannot succeed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Zeta] Log node-offline task failures as warning [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11856:
URL: https://github.com/apache/seatunnel/pull/11856#issuecomment-6138789823
@SEZ9 I pushed `eed3d9f2f41` to close the remaining F1/F2 items from https://github.com/apache/seatunnel/pull/11856#issuecomment-5852104617.
- **F1 lifecycle assumption:** `SeaTunnelServer.java:280-285` now documents that the marker write relies on Hazelcast 5.1 `LifecycleServiceImpl#shutdown()` publishing `SHUTTING_DOWN` before distributed-object proxies stop accepting the write, and explicitly asks that this ordering be re-verified on a Hazelcast upgrade.
- **F2 restore guard:** `CoordinatorService.java:1065-1117` now encloses the entire active-job restore phase in `try/finally`. `restoringRunningJobsFromMasterSwitch` is reset after success and also after worker-wait interruption, future failure, or any other exceptional exit. This closes the bug where every later marker was retained until its five-minute TTL.
- **F2 regression coverage:** `CoordinatorServiceTest#testGracefulMarkerClearsAfterActiveJobMasterSwitchRestore` (`:305-391`) runs a real two-node cluster, submits a live streaming job, switches the active master, waits for the job to be running on the new master, then verifies a subsequent graceful-removal marker is removed immediately rather than retained.
- **Non-clearing restore read:** `PhysicalVertex.java:273-278` now states explicitly that the restore-path read is intentionally non-clearing and why the membership callback owns the conditional clear.
The earlier F3-F8 fixes remain present: member-removal failures still carry `JobException`, the marker has Hazelcast entry TTL plus conservative timestamp validation, WARN/ERROR classification is typed rather than message-matched, the offline message has one producer, and both EN/ZH recovery docs are present.
Formatting verification on `eed3d9f2f41`: module-scoped `spotless:apply`, `spotless:check`, and `git diff --check` passed. Per repository policy I did not run local compilation/tests; the new GitHub CI run is the runtime verification. The prior head's only completed failing test was unrelated `PayPalClientTest.closeWakesRetryWait` on Windows; the remaining long-running jobs were cancelled at the workflow deadline. I will not claim the requested per-method evidence for `testGracefulShutdownPublishesMemberRemovalMarker` until the new CI run has actually executed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Zeta] Log node-offline task failures as warning [seatunnel]
Posted by "davidzollo (via GitHub) via commits" <co...@seatunnel.apache.org>.
davidzollo commented on PR #11856:
URL: https://github.com/apache/seatunnel/pull/11856#issuecomment-6703590171
<!-- daniel-pr-followup-20261107 -->
Yes, contributing to Apache SeaTunnel.
Pushed `b32e5360d881cc9cdf889a9a44370767129babf3` by normal fast-forward push. Fixed CoordinatorServiceTest fixture compilation: MembershipServiceEvent.getMember returns MemberImpl, so the mock now uses that concrete return type. Production behavior and assertions are unchanged.
Validation: affected-module Spotless and diff whitespace checks passed; an independent maintainer source review of these deltas completed. No local compile, unit, integration, E2E, Docker or service execution was performed under the repository validation policy. No engine UI change or existing runtime artifact was used. The current-head [fork Build](https://github.com/DanielLeens/seatunnel/actions/runs/42346454327) was queued at the post-push inspection; this update does not claim runtime correctness or merge readiness. This supersedes earlier verification statements for the previous head.
Existing connector registrations, configuration defaults, public contracts and prior test assertions were retained. Upstream-only POM changes in the approved conflict merges match dev exactly; no new dependency edits were made. Human review and any remaining explicit coverage/acceptance gaps are separate gates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Zeta] Log node-offline task failures as warning [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11856:
URL: https://github.com/apache/seatunnel/pull/11856#issuecomment-5433080579
CI is green on `94eaf415` — all 87 Build jobs passed. The only failure in the first round was `CouchbaseIT.testFakeSourceToCouchbaseSink` (Flink-engine container variant) hitting a Couchbase test-container auth/readiness timeout (`WaitUntilReady` stuck in `WAIT_FOR_CONFIG`); it passed on a failed-jobs-only rerun without any code change, and the Zeta variants of the same test were green in both rounds, so it was environmental, not related to this Zeta-only diff. From my side this is now ready for a fresh maintainer review — @SEZ9 the re-review request stands whenever you have time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Zeta] Log node-offline task failures as warning [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11856:
URL: https://github.com/apache/seatunnel/pull/11856#issuecomment-5421604523
Thanks @DanielLeens for the fresh pass over `06344bd0` — no apology needed; races like this are easy to miss until the interleavings are traced deliberately.
I take the new concurrency concern seriously: if the error payload and its graceful-removal classification can be updated as a non-atomic pair, a genuine failure could be misclassified and downgraded to WARN. That's effectively a concurrency variant of the log-level classification concern you already raised, so I'll treat it as part of closing that finding.
Two things I need before I can fix and verify this properly:
1. Your review text appears to have been cut off mid code snippet — could you post the rest, especially the exact interleaving you traced? I'd like to reproduce it in a test before claiming a fix works.
2. You mentioned this round also covers the current CI failure, but that part didn't come through either. Can you share what you saw failing?
Glad to hear the 8 items from the earlier review (`c801f463`) independently re-verify as resolved on the current head. Once the race details and CI info are in, I'll push a follow-up commit.
<!-- streview-comment:571 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Zeta] Log node-offline task failures as warning [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11856:
URL: https://github.com/apache/seatunnel/pull/11856#issuecomment-5747479000
Thanks for the detailed follow-up on `dc2c64103c`.
**F1** – Moving the marker write out of `ManagedService.shutdown()` and onto the `LifecycleListener` path (`SHUTTING_DOWN` → `markLocalGracefulMemberRemoval()`) is the right fix for the PASSIVE-node problem, and having `ClusterFaultToleranceIT.testGracefulShutdownPublishesMemberRemovalMarker` drive it through two real members with `node1.shutdown()` is what I was asking for. Two things before I close this one:
1. As you note, the evidence from fork run `35240527669` is a class-level result for `ClusterFaultToleranceIT`, and the new test passing is an inference. Could you share per-method output (from CI or a local run of that single test) so we have direct confirmation?
2. You mention the SIGTERM / JVM shutdown-hook path is covered by reading the Hazelcast 5.1 sources rather than by a test. I'm fine not adding an IT for the hook itself, but please add a short comment near `markLocalGracefulMemberRemoval()` noting that the marker depends on `LifecycleServiceImpl.shutdown` firing `SHUTTING_DOWN` on the hook path, so the assumption is visible to whoever bumps Hazelcast next.
**F4** – Your comment appears to have been cut off right after "Eviction is a Hazelcas…", so I only have the first sentence of the TTL-versus-timestamp explanation. Could you re-post the rest? In particular, how the TTL on the `put` and the `isGracefulMemberRemovalMarkerValid` timestamp check divide responsibility, and whether the timestamp check still depends on cross-node clock agreement.
**F2, F3, F5, F6, F7, F8** – I don't see these addressed in this comment. If they were handled in `dc2c64103c`, a brief per-finding pointer is enough; otherwise please let me know which you intend to fix in this PR and which you'd rather defer, so we can converge on the remaining scope.
<!-- streview-comment:1187 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Zeta] Log node-offline task failures as warning [seatunnel]
Posted by "DanielLeens (via GitHub) via commits" <co...@seatunnel.apache.org>.
DanielLeens commented on PR #11856:
URL: https://github.com/apache/seatunnel/pull/11856#issuecomment-6879358528
### Implementation Notes and Trade-offs
Yes, contributing to Apache SeaTunnel. Follow-up on head `1f003592570833eb48b2a7edad0830e960e1c462`.
The regression test introduced by b32e5360d88 imported a Hazelcast class from the wrong package. The test now imports the concrete type returned by MembershipServiceEvent#getMember(). The mock and assertions are unchanged; no production change or broader fixture rewrite was needed. Current-head unit and engine integration results are required.
The focused Java file passed Spotless using the repository's version and Java rules (isolated formatter project, without the Maven reactor) and the staged diff passed the whitespace check. Independent source review found no blocker in this correction. No local compile, runtime, E2E, or UI validation was run. GitHub CI on this head is required before runtime verification can be claimed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Zeta] Log node-offline task failures as warning [seatunnel]
Posted by "DanielLeens (via GitHub) via commits" <co...@seatunnel.apache.org>.
DanielLeens commented on PR #11856:
URL: https://github.com/apache/seatunnel/pull/11856#issuecomment-6538592838
Merged `dev` (`3ac2d1cec6a`) into this branch with a plain merge commit (no rebase, no force push). New head: `408723b36e2` (previous head `eed3d9f2f4`). Only formatting was run locally (module-scoped `spotless:apply`); `git diff origin/dev --name-status` lists only this PR's own 11 files and no deletions.
Conflicts and resolution:
- `deployment-seatunnel-master.yaml`, `deployment-seatunnel-worker.yaml`, `seatunnel-cluster.sh`: `dev` (#12710) already switched the foreground start to `exec java` and the K8s command to exec form, which is the same intent as this PR's launch-path bullet. Took `dev`'s version, so these three files no longer differ from `dev`. As a consequence this PR's own string-matching `testClusterScriptExecsForegroundServer` in `SeaTunnelClusterScriptTest.java` was dropped: `dev`'s `testClusterScriptExecsServerJvmInForeground` already asserts the same behaviour (JVM parent pid equals the script pid) and is stricter.
- `PhysicalVertexTest.java` (add/add): `dev` (#12042) added a test class with the same name. Merged into one class: `dev`'s six tests and helpers are unchanged, this PR's graceful-removal tests and helpers are appended, and the class Javadoc covers both.
- `CoordinatorService.java`, `SeaTunnelServer.java`, `PhysicalVertex.java`, `ClusterFaultToleranceIT.java` and both state-storage-and-recovery docs merged without textual conflict. I re-read the merged `restoreAllRunningJobFromMasterNodeSwitch` path: the `restoringRunningJobsFromMasterSwitch` try/finally reset and `dev`'s zombie pre-filter are both intact; the `initCoordinatorService()` reset is intact.
Compile and tests are left to the fork CI run per the validation policy (queued at the time of writing, no result claimed): https://github.com/DanielLeens/seatunnel/actions/runs/41165429981
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Zeta] Log node-offline task failures as warning [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11856:
URL: https://github.com/apache/seatunnel/pull/11856#issuecomment-5551902649
Merged `dev` into the branch (`e6e243744e3`, GitHub "update branch" merge commit, no manual changes) and restarted CI.
Why: on `9e433426e49` the only red left after reruns was `all-connectors-it-7 (8)` — `PostgresCDCIT.testPostgresCdcSnapshotOnlyAndCommittedOffsetStartupModes` timing out at `PostgresCDCIT.java:603` (`expected: <1> but was: <0> within 2 minutes`), 3 out of 3 attempts on JDK 8. That line on this branch was still `await().atMost(120, TimeUnit.SECONDS)`; the branch was 134 commits behind `dev` and did not have #11644 ("Give PostgresCDCIT post-restore assertions their own budget", `577032fb5a7`), which replaced it with the larger `RESTORE_ASSERT_TIMEOUT_MILLIS`. The underlying race is the open #11847 (fix in #11864, not merged yet), so a fourth rerun on the old budget was not a real fix; picking up `dev`'s budget is.
Everything else on `9e433426e49` was green after reruns (`doris-connector-it (8)` was a one-off Doris BE readiness failure: `NoHttpResponseException` on `172.18.0.2:8040`, passed on rerun; `all-connectors-it-7 (11)` passed on rerun). The three-dot diff against `dev` is unchanged by the merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Zeta] Log node-offline task failures as warning [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11856:
URL: https://github.com/apache/seatunnel/pull/11856#issuecomment-5807035361
Thanks for re-tracing the diff and runtime path against `dc2c64103c` rather than only extending the earlier conclusions. One note: the comment as it appears on the thread ends mid-sentence at "the TTL/clock-skew rule", so nothing after section 1.1 (including the "Issue 4" you refer to) is visible here. If it is a rendering artifact again, a short inline restatement would be enough. Going point by point against the open findings:
**F1 (marker published during shutdown while the node is already PASSIVE)** — moving the publish into `stateChanged()` on `SHUTTING_DOWN`, with the stale-marker clear on `STARTED`, is the right shape and addresses the core concern. Two remaining asks: (1) please make sure the code comment about the `SHUTTING_DOWN`/Hazelcast-version dependency lands in the diff, since the feature relies on IMap writes still being accepted in that state; (2) the class-level `Tests run: 10, Failures: 0, Errors: 0` is good, but please paste or link the per-method line for the new graceful-shutdown test so we can see it ran rather than being skipped.
**F2 (destructive consume before processing / failover loses classification)** — reading the marker once per removal event and sharing one boolean across affected tasks helps consistency, but it doesn't tell me whether the entry is still removed before task-failure processing completes. Please confirm whether the marker is now read non-destructively and left to expire, or describe how a master failover mid-processing keeps the graceful classification.
**F4 (no eviction on `engine_gracefulMemberRemoval`; manual TTL depends on clock agreement)** — the relevant text is not visible on this thread. Please restate briefly: is expiry enforced by map-level TTL/eviction configuration, or still by a manual timestamp comparison on the reader side?
**F3 / F6 (JobException replaced by a bare String in the task state and in the persisted/REST error payload)** — not addressed in the visible portion. Please confirm whether the graceful path still carries a throwable with the original stack trace, or whether the "state handling unchanged" claim in the description needs revising.
**F5 / F8 (free-text message matching for the WARN downgrade; template defined twice)** — also not covered in what's visible. If classification is now derived from the marker lookup rather than from matching message text, please say so and point to where the two definitions were collapsed into one.
**F7 (docs)** — please confirm whether a `docs/` entry describing the marker map and the WARN downgrade of node-offline failures is part of the current head.
Once the missing sections are re-posted or answered inline, I can close out the remaining items.
<!-- streview-comment:1283 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Zeta] Log node-offline task failures as warning [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11856:
URL: https://github.com/apache/seatunnel/pull/11856#issuecomment-5812765027
Hi @SEZ9, thanks for going through this so carefully again.
On the rendering issue: I already flagged this once for the F4 explanation at the top of my last review (`dc2c64103c`, review id 5289961888), and it looks like it's happened again, this time apparently cutting off partway through section 1.1. I just re-pulled that review body directly through the GitHub API to double-check: it is **not** actually truncated server-side — the stored comment is about 22.6 KB / 186 lines and continues well past the "TTL/clock-skew rule" sentence, through the Key findings, the runtime-path trace, the full Issue 1 write-up, sections 1.2-1.4, all four numbered issues, sections 2-5, and the merge recommendation. So this looks like a GitHub UI/client rendering artifact on long comments rather than a real content gap on my end. Restating the parts you couldn't see, point by point:
**Issue 4 (the item behind F1's second ask)** — already logged as a formal issue in that review, not something missing:
> `markLocalGracefulMemberRemoval()` should document its Hazelcast-version dependency. Location: `SeaTunnelServer.java:280-290`. The correctness of the whole feature depends on `LifecycleServiceImpl.shutdown()` firing `SHUTTING_DOWN` before the node stops being able to serve distributed-object proxies — established only by reading the Hazelcast 5.1 source, not asserted anywhere near this method. Severity: Low. Raised by another reviewer: Yes (you, comment 5747479000, F1 item 2).
To be precise about status: this is a recommended fix from the review, not yet landed in the diff — `dc2c64103c` hasn't changed since my 09-18 comment, so no comment has been added near that method yet. It's tracked as non-blocking (Low), same tier as Issue 2 (sync IMap calls / log noise) and Issue 3 (TTL-vs-heartbeat docs gap).
**F1, per-method test confirmation** — still genuinely open; I'd rather say that plainly than restate the class-level number as if it settles it. What I have is `Tests run: 10, Failures: 0, Errors: 0, Skipped: 2` for the whole `ClusterFaultToleranceIT` class from fork run `35240527669`. I have not found a per-method line for `testGracefulShutdownPublishesMemberRemovalMarker` specifically in that log — I said the same in my 09-19 comment and don't have a stronger source today. Honest status: class-level pass confirmed, method-level still unconfirmed.
**F2, non-destructive read / clear timing** — unchanged from my 09-19 answer and still correct: `CoordinatorService.java:2185` does a non-destructive `IMap.get`, and the value-conditional `remove` at `:2207` only runs after the propagation loop finishes (`:2222-2238`) and only when `canClearGracefulMemberRemovalMarker` allows it (`:2239-2245`). So the marker is not removed before task-failure processing completes for the event that read it. What my latest pass adds on top is Issue 1: `restoringRunningJobsFromMasterSwitch` is reset only inside `initCoordinatorService()` and never after a restore actually completes, so after the first master-switch restore with active jobs, `canClearGracefulMemberRemovalMarker` never returns `true` again for the rest of that master's term — the explicit clear effectively becomes dead code and everything falls back to the 5-minute TTL. That's a real, currently open gap (High), distinct from the non-destructive-read question, which is fine on its
own.
**F3/F6, F5/F8, F7** — unchanged from the 09-19 table, nothing new: the `JobException` payload shape is still pinned by a dedicated test (F3/F6, resolved); there's no `Pattern`/`.matches(` string-matching left in `PhysicalVertex.java` and classification is a single typed boolean from one producer helper (F5/F8, resolved); both `docs/en` and `docs/zh` `state-storage-and-recovery.md` carry the new marker/TTL/WARN-ERROR section (F7, resolved).
Net status: the only thing blocking merge from my side right now is Issue 1 (the never-reset flag) — a small, well-scoped fix (reset it once the restore future completes) plus a regression test. Issues 2-4 are non-blocking follow-ups. No new commit has landed on `dc2c64103c` since my 09-18/09-23 passes, so there's nothing else on this head to re-verify yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Zeta] Log node-offline task failures as warning [seatunnel]
Posted by "SEZ9 (via GitHub) via commits" <co...@seatunnel.apache.org>.
SEZ9 commented on PR #11856:
URL: https://github.com/apache/seatunnel/pull/11856#issuecomment-6553000321
Thanks for merging `dev` (`3ac2d1cec6a`) into this branch as `408723b36e2`. The conflict resolution as described sounds reasonable: taking `dev`'s `exec java` launch path and dropping `testClusterScriptExecsForegroundServer` in favour of `dev`'s stricter test, and combining the two `PhysicalVertexTest.java` classes into one. I'll wait for the fork CI run (41165429981) for compile/test status.
This update covers the merge itself; it doesn't say whether any of the previously raised findings changed between `eed3d9f2f4` and `408723b36e2`. Before I re-review, could you state, per finding, whether it was addressed or is still open:
- **PR11856-F1 (HIGH)** – publishing the graceful-shutdown marker from `ManagedService.shutdown()`, where the node is already PASSIVE and IMap writes can be rejected. This is the blocker: if the marker write silently fails, the WARN-downgrade no-ops in real graceful shutdowns. Has the marker publication been moved earlier in the shutdown sequence, or is there a test showing the marker is reliably present after a graceful shutdown?
- **PR11856-F2** – the marker is destructively consumed before task-failure processing and has no physical expiry; master failover mid-processing loses the graceful classification.
- **PR11856-F3 / PR11856-F6** – the graceful path replaces the `JobException` throwable in `TaskExecutionState` with a plain String, which contradicts the "state handling unchanged" claim and changes the error payload users see in job history / REST.
- **PR11856-F4** – `engine_gracefulMemberRemoval` entries have no eviction and the manual TTL check depends on cross-node clock agreement.
- **PR11856-F5** – classifying log level by free-text message match means any failure that mimics the offline template gets downgraded to WARN.
- **PR11856-F7** – you mention both state-storage-and-recovery docs are part of this PR; do those changes describe the new marker IMap and the WARN downgrade of node-offline failures?
- **PR11856-F8** – the offline-message template exists twice (format string in `CoordinatorService.java`, regex in `PhysicalVertex.java`) and will drift silently.
If any of these are intentionally deferred, please say so explicitly so we can decide whether they block this PR or can be follow-ups. For F1 in particular I'd like to see it resolved (or shown not to be a problem) before merge.
<!-- streview-comment:2192 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Zeta] Log node-offline task failures as warning [seatunnel]
Posted by "DanielLeens (via GitHub) via commits" <co...@seatunnel.apache.org>.
DanielLeens commented on PR #11856:
URL: https://github.com/apache/seatunnel/pull/11856#issuecomment-6646505520
The CI fact cited previously has changed: exact-head fork Build 41165429981 failed. The Windows JDK 17 unit log identifies a shared test-compilation error at CoordinatorServiceTest.java:379: thenReturn(Member) cannot satisfy the method's MemberImpl return type. The unit matrices and many IT prerequisites therefore fail before validating their tests. Please align the fixture/mock with the concrete return contract and rerun compilation before rerunning the broad matrix; do not treat dozens of downstream red lanes as independent connector regressions. This observed compile failure blocks validation and supersedes the earlier source-clear/unfinished-CI state. No upstream fix has been established for this exact compile mismatch, so a blind sync is not the proposed remedy.
CI reference: [Build 41165429981](https://github.com/DanielLeens/seatunnel/actions/runs/41165429981).
This updates CI/merge facts on the already-reviewed head; it is not another full source review. No local runtime testing was performed. Daniel has READ permission; effective maintainer approval/dismissal remains separate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org