You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "nielifeng (via GitHub)" <gi...@apache.org> on 2026/07/27 02:32:49 UTC
[PR] [Fix][Zeta] Offload report metrics operations [seatunnel]
nielifeng opened a new pull request, #11564:
URL: https://github.com/apache/seatunnel/pull/11564
### Purpose of this pull request
Fixes #10997.
`ReportMetricsOperation` performs the distributed metrics IMap update
synchronously. Running it on Hazelcast generic-operation threads lets metrics
work contend with checkpoint and task lifecycle control operations under heavy
reporting load.
This patch makes `ReportMetricsOperation.call()` return Hazelcast's explicit
`Offload` status and schedules the metrics update on the built-in
`OFFLOADABLE_EXECUTOR`. The operation response is sent only after the metrics
update finishes, so callers retain the existing invocation completion
semantics. The metrics payload, serializer ID, storage, and
`SeaTunnelServer.updateMetrics()` behavior are unchanged.
### Does this PR introduce _any_ user-facing change?
Yes. Internal metrics reports no longer execute on Hazelcast generic-operation
threads, reducing the risk that metrics reporting delays checkpoint and batch
job completion control operations. There is no public API or configuration
change.
### How was this patch tested?
- Added `ReportMetricsOperationTest` with a real local Hazelcast member to
verify the operation leaves generic-operation threads, runs through the
offloadable executor, keeps the invocation future incomplete until the
metrics update finishes, and propagates update failures.
- Verified the existing metrics IMap integration path still stores metrics
across the configured partitions.
```shell
./mvnw -o -pl seatunnel-engine/seatunnel-engine-server \
-Dtest=ReportMetricsOperationTest test
./mvnw -o -pl seatunnel-engine/seatunnel-engine-server \
-Dtest=CoordinatorServiceTest#testMetricsImapSizeWithPartitionConfig test
```
Both focused tests pass on the latest `dev` branch.
### Check list
* [x] No new Jar binary package is added.
* [x] Documentation is not required because there is no new feature or user configuration.
* [x] No incompatible change is introduced.
* [x] Connector-specific checklist items are not applicable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Offload report metrics operations [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11564:
URL: https://github.com/apache/seatunnel/pull/11564#issuecomment-5358316750
Thanks @DanielLeens for the detailed re-verification, and @nzw921rx for the +1.
The diff evidence is exactly what I needed: confirming that the two files in this PR's diff (`ReportMetricsOperation.java` and `ReportMetricsOperationTest.java`) are byte-for-byte identical between `2553537c465f`, `e22ffca5e440`, and the current head `e01712f9`, with the only new commit being the merge of `dev`, means there is no new logic delta to review. Since the merge picked up no changes to the PR's files, my earlier ask to merge exactly `e22ffca5e` is satisfied in substance — we'd now merge `e01712f9`, which carries the identical source.
Two concrete remaining asks before merge:
1. Please confirm no further pushes are planned on this head so we merge exactly `e01712f9`.
2. The non-blocking test-stability note on `ReportMetricsOperationTest.shouldKeepRemoteGenericOperationsResponsiveWhileMetricsWaitForOffloadExecutor` still needs a follow-up tracker — a short summary of the suggested improvement in a new issue would be enough so it doesn't get lost after merge.
Once those are in place, I'll proceed with the merge. Thanks both for the careful reviews on this one.
<!-- streview-comment:393 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Offload report metrics operations [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11564:
URL: https://github.com/apache/seatunnel/pull/11564#issuecomment-5363691441
Thanks @SEZ9. On the two asks:
1. **Head SHA** — I can confirm the merge-mechanics half of this: `headRefOid` for this PR is still `e01712f9` right now, unchanged since it was pushed on 2026-08-19, and the two files that make up this PR's actual diff (`ReportMetricsOperation.java`, `ReportMetricsOperationTest.java`) are still byte-for-byte identical to the head I approved on 08-04. Whether any *further* push is planned is really a question for @nielifeng to answer directly, since that's outside what I can attest to from the repo state alone.
2. **Correction to my own last review** — I need to flag something before this merges. In my 08-19 comment I described the fork `Build` run on `e01712f9` as still `in_progress` and treated CI-completeness as the only remaining gate. That run has since finished, and it is now **failing**, not green. I went and checked the actual failing jobs so this doesn't block on a guess:
- `unit-test (11, windows-latest)`
- `engine-v2-it (8, ubuntu-latest)` — multi-node `ClusterFaultToleranceIT` restore/master-down scenarios and a `RestApiIT` heartbeat-timeout pattern
- `all-connectors-it-7 (11, ubuntu-latest)`
- `doris-connector-it (8, ubuntu-latest)` — concretely, `DorisErrorIT.testDoris:82 expected: <true> but was: <false>`
None of these touch `ReportMetricsOperation`/`ReportMetricsOperationTest` at all (Doris connector IT and Windows unit-test are obviously unrelated modules; the `engine-v2-it` failures are fault-tolerance/REST integration tests, not anything in the metrics-offload call path), and the PR's own diff is unchanged from the head that passed CI twice before. So this looks like environment-level flakiness in unrelated modules rather than a regression introduced here — but it does mean the `Build` check is factually red on `e01712f9` right now, and `mergeStateStatus` is `BLOCKED`. I'd suggest re-running just the failed jobs (`gh run rerun <run-id> --failed` on the fork run) before merging, rather than treating CI as already satisfied.
On the test-stability tracker: agreed a short follow-up issue for the `shouldKeepRemoteGenericOperationsResponsiveWhileMetricsWaitForOffloadExecutor` timing-centralization note (mine and yours from 07-28/08-01) is the right way to keep it from getting lost post-merge — happy to see that filed, no objection from my side either way on who opens it.
No new source-level issue from me this round; my source conclusion (APPROVED, one Low-severity non-blocking item) still stands. The only thing that changed is the CI fact above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Offload report metrics operations [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11564:
URL: https://github.com/apache/seatunnel/pull/11564#issuecomment-5389993355
Thanks @SEZ9 — agreed on all three points, and nothing new from me on the source side.
Just checked live again: `Build` on `e01712f9` is still `failure` for the same jobs we already discussed (`unit-test (11, windows-latest)`, `engine-v2-it (8, ubuntu-latest)`, `all-connectors-it-7 (11, ubuntu-latest)`, `doris-connector-it (8, ubuntu-latest)`), none of which touch `ReportMetricsOperation`/`ReportMetricsOperationTest`. So the rerun still hasn't happened yet — that action item is still with @nielifeng.
To be precise about what "nothing further from review side" means: my APPROVED conclusion from 08-04 stands, reaffirmed multiple times since (most recently 08-22) with the diff confirmed byte-for-byte unchanged each time. Once the failed jobs are rerun and `Build` comes back green on this same head, there's no additional review step needed from me before merge.
On the tracking issue for the `shouldKeepRemoteGenericOperationsResponsiveWhileMetricsWaitForOffloadExecutor` timeout-centralization note — appreciate you taking that on post-merge, that keeps it from getting lost.
@nielifeng — whenever you get a chance, a rerun of just the failed jobs on `e01712f9` and a note here once green would unblock this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Offload report metrics operations [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11564:
URL: https://github.com/apache/seatunnel/pull/11564#issuecomment-5385641459
Thanks @DanielLeens for the detailed follow-up and for correcting your earlier read of the `Build` run — much appreciated.
Agreed on all points:
1. **Head SHA** — good, `e01712f9` unchanged since 2026-08-19 and byte-for-byte identical to the head you approved on 08-04. That matches my expectation; no further push is needed from a review standpoint.
2. **CI status** — I see the same thing: `Build` on `e01712f9` is `failure` (https://github.com/apache/seatunnel/runs/95934660228). Given the failing jobs (`unit-test (11, windows-latest)`, `engine-v2-it (8, ubuntu-latest)`, `all-connectors-it-7 (11, ubuntu-latest)`, `doris-connector-it (8, ubuntu-latest)`) don't touch `ReportMetricsOperation`/`ReportMetricsOperationTest` and this head passed CI twice before, I agree this looks like environment-level flakiness rather than a regression from this PR. That said, we shouldn't merge on a red check — the rerun of the failed jobs (`gh run rerun <run-id> --failed` on the fork run) is the remaining blocker. Since your access is read-only, that has to come from the PR author; asking them to trigger it and report back here once it's green.
3. **Follow-up tracker** — agreed a short issue for the `shouldKeepRemoteGenericOperationsResponsiveWhileMetricsWaitForOffloadExecutor` timeout-centralization note is the right way to keep it from being lost. To avoid it stalling on "whoever gets to it first," I'll file it after merge and link it back to this thread.
So concrete remaining asks:
- PR author: rerun the failed `Build` jobs on `e01712f9` and confirm green here.
- Nothing further from review side — your APPROVED conclusion stands and the diff is unchanged, so once CI is green this is ready to merge.
<!-- streview-comment:489 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Offload report metrics operations [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11564:
URL: https://github.com/apache/seatunnel/pull/11564#issuecomment-5165359994
Thanks for the extra current-head pass.
On this unchanged head, I do not have a separate new source-side blocker to add beyond the current gate state I already called out. The offload direction itself still looks reasonable to me; the thing that changed after my earlier note was the CI fact, so for this revision the next step is still to anchor follow-up on the latest failing check context rather than asking for unrelated source churn.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Offload report metrics operations [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11564:
URL: https://github.com/apache/seatunnel/pull/11564#issuecomment-5366243678
Thanks @DanielLeens for the thorough re-verification, and @nzw921rx for the earlier approval.
Summarizing where we stand on the offload change:
- The source at head `e01712f9` is confirmed byte-for-byte identical to the previously approved heads (`2553537c465f`, `e22ffca5e440`) for the two files in the diff, with the only new commit being the merge from `dev` that doesn't touch `ReportMetricsOperation.java` or `ReportMetricsOperationTest.java`. So there's no new logic to review — the `Offload` / `OFFLOADABLE_EXECUTOR` direction remains approved from the source side.
- On the CI status you flagged: given the failing jobs (`unit-test (11, windows-latest)`, `engine-v2-it (8, ubuntu-latest)`, `all-connectors-it-7 (11, ubuntu-latest)`, `doris-connector-it (8, ubuntu-latest)`) don't intersect the metrics-offload call path and the same diff passed CI twice before, I agree this reads as unrelated flakiness rather than a regression here. That said, we shouldn't merge on a red `Build`.
Concrete remaining asks before merge:
1. Please re-run just the failed jobs on `e01712f9` as suggested (`gh run rerun <run-id> --failed`) and report back once the `Build` check is green.
2. Please file the short follow-up issue for the `shouldKeepRemoteGenericOperationsResponsiveWhileMetricsWaitForOffloadExecutor` timing-centralization note so the Low-severity non-blocking item is tracked post-merge — either of you opening it works for me.
Once CI is green on the current head, this is good to go from my side. Thanks both!
<!-- streview-comment:422 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Offload report metrics operations [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11564:
URL: https://github.com/apache/seatunnel/pull/11564#issuecomment-5117039797
Quick follow-up after the later review activity: I do not see a new Daniel-side source blocker on this head.
The later review notes are useful follow-up suggestions, and the current `Build` check is green. From Daniel's side, this is now mainly waiting on the normal write-capable maintainer review/merge gate rather than a new code change request from me.
Thanks for continuing to improve the Zeta metrics path. This is a good direction for keeping metrics reporting away from checkpoint and task-lifecycle control threads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Offload report metrics operations [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11564:
URL: https://github.com/apache/seatunnel/pull/11564#issuecomment-6396815799
Thanks for consolidating this, @SEZ9. These overlap the executor/backlog, asynchronous response/error contract and test-seam concerns already discussed here, so I agree with keeping one existing list rather than creating duplicate findings.
Please address the bounded engine-owned executor and accurate `Offload` Javadoc first, while preserving response-failure visibility and Hazelcast's handling of fatal errors. The test should use an injectable executor seam instead of reaching into a private Hazelcast field. No new commit is present yet, so I cannot mark these code-side concerns cleared or treat a CI rerun as a substitute for the fix. I will wait for the updated implementation and review the complete diff on that head.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Offload report metrics operations [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11564:
URL: https://github.com/apache/seatunnel/pull/11564#issuecomment-5409366827
Thanks @DanielLeens for re-checking the status.
1. **CI rerun:** Confirming the `Build` check on `e01712f9` is still failing (run 95934660228) and the rerun of the failed jobs hasn't been triggered yet. As you noted, that rerun needs to come from the PR author's side, so it remains the outstanding action here. Agreed that the failing jobs don't touch this PR's diff, so rerunning just the failed jobs is the right approach.
2. **Tracking issue:** The follow-up issue for the `shouldKeepRemoteGenericOperationsResponsiveWhileMetricsWaitForOffloadExecutor` timeout-centralization note will be filed post-merge, as already agreed.
So the only remaining gate before merge is a green `Build` on this same head.
<!-- streview-comment:529 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Offload report metrics operations [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11564:
URL: https://github.com/apache/seatunnel/pull/11564#issuecomment-6379897794
Thanks for keeping this up to date with `dev`. As noted on the thread, the only change between `e01712f9` and `d71ee0a433fb` is the merge commit, and `ReportMetricsOperation.java` / `ReportMetricsOperationTest.java` are identical between the two, so the previously raised points are still open.
Here is what remains, in priority order:
1. **Executor sizing / backlog (PR11564-F1, MEDIUM).** `Offload.start()` queues `run()` onto Hazelcast's shared `OFFLOADABLE_EXECUTOR`, which is core-sized with an unbounded queue and is also used by Hazelcast's own offloaded map operations. Under the heavy-reporting load this PR targets, callers can time out and re-send while the backlog keeps growing. Please either use a dedicated, bounded executor for metrics updates, or explain in the PR why the shared pool is acceptable here.
2. **Javadoc on the `call()` override (PR11564-F2, MEDIUM).** This is the first `Offload` operation in Zeta, and the current one-line comment is misleading. Please replace it with a Javadoc that states the actual contract: the generic-operation thread returns immediately, `run()` executes on the offload executor, and the response (success or failure) is sent manually from that thread.
3. **Success-path `sendResponse(null)` failure (PR11564-F3, LOW).** If `sendResponse(null)` throws inside the offloaded runnable, nothing is logged through the operation, no failure response is attempted, and the caller waits for the invocation timeout. Please at least log via the operation's logger so this is diagnosable.
4. **`catch (Throwable t)` swallowing `Error`s (PR11564-F4, LOW).** `OutOfMemoryError` and other `Error`s are currently routed into the normal failure-response path, bypassing Hazelcast's `OutOfMemoryErrorDispatcher` hook that the generic-operation runner applied before this change. Please narrow the catch or explicitly inspect/rethrow `OutOfMemoryError`.
5. **Reflection on `Offload.executionService` in the test (PR11564-F5, LOW).** `shouldAttemptSuccessResponseOnlyOnceWhenResponseDeliveryFails` reaches into a Hazelcast internal field by name, which is brittle across Hazelcast upgrades. If a dedicated executor is introduced for item 1, injecting it would let this test avoid reflection.
Items 1 and 2 are what I'd like resolved before approval; 3–5 are small and could go into the same push. If you disagree with any of these, a short note on the thread is fine and we can discuss.
<!-- streview-comment:2000 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Offload report metrics operations [seatunnel]
Posted by "nielifeng (via GitHub)" <gi...@apache.org>.
nielifeng commented on PR #11564:
URL: https://github.com/apache/seatunnel/pull/11564#issuecomment-5088534325
Follow-up validation after review findings:
- Fixed the success-response exception boundary so a failed `sendResponse(null)` is never followed by a second failure response.
- Added a focused red/green regression: the previous implementation fails with `expected: <1> but was: <2>`; the updated implementation passes.
- Strengthened the remote offload test with 4 target generic-operation threads while 3 are deliberately occupied. With the offload executor saturated, another remote generic operation still completes through the only available generic thread. This is local test isolation only; the real cluster was never set back to 1 thread.
- Fresh local result: `ReportMetricsOperationTest` 3/3 passed, failures/errors/skips 0/0/0.
- Independent code review found no P0-P3 issue.
Second-cluster load evidence remains: patched 60/min x 20m completed 1200/1200; patched generic=4 60/min x 10m completed 600/600; metrics failure/interrupted 0/0; no failed jobs; slots fully recovered. The cluster was rolled back to the original JAR and generic threads=50 after validation. The final response-boundary change affects only response-delivery failure handling and is covered by the new local red/green test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Offload report metrics operations [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11564:
URL: https://github.com/apache/seatunnel/pull/11564#issuecomment-5379386710
Thanks @SEZ9 — quick follow-up so this doesn't stall.
I just re-checked live: the `Build` check on the current head `e01712f9` is still `failure` (https://github.com/apache/seatunnel/runs/95934660228), so it looks like the rerun hasn't happened yet. Since Daniel's permission on this repository is read-only, I can't trigger that rerun myself — ask #1 (`gh run rerun <run-id> --failed` on the fork's run behind this pointer check) needs @nielifeng to do it and report back once it's green.
On ask #2 (the follow-up issue for the `shouldKeepRemoteGenericOperationsResponsiveWhileMetricsWaitForOffloadExecutor` timeout-centralization note): no objection from me on who files it, same as I said before — happy to leave it to whoever gets to it first among us.
No new source-side finding from me on this pass; my APPROVED conclusion from 08-04 (reaffirmed most recently on 08-19, and confirmed the diff is still byte-for-byte unchanged) still stands. Just waiting on the CI rerun and the tracking issue above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Offload report metrics operations [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11564:
URL: https://github.com/apache/seatunnel/pull/11564#issuecomment-6392877830
Following up on head `d71ee0a433fb`: the five points from the previous review still stand, and there is agreement on the thread that none of them are contradicted. To keep the list in one place, here is what I'd like to see in the next push:
**Must-fix before approval**
1. **Executor ownership / backlog (F1).** `ReportMetricsOperation` currently offloads onto Hazelcast's shared, core-sized `OFFLOADABLE_EXECUTOR` with an unbounded queue. Under the heavy-reporting load this PR is meant to handle, the backlog can grow without bound while callers time out and re-send, and it competes with Hazelcast's own offloaded map operations. Please move this to a dedicated, bounded executor owned by the engine (or explain on the thread why the shared executor is acceptable if you disagree — a short note is fine).
2. **Javadoc for the `call()` override (F2).** The single-line comment is misleading about what actually happens. Please replace it with a Javadoc that states the real `Offload` contract: that `call()` returns an `Offload`, that the work and the response are delivered asynchronously from the executor, and why this is needed for metrics reporting.
**Can ride in the same push**
3. **Success-path `sendResponse(null)` failure (F3).** If `sendResponse(null)` throws, the exception escapes the runnable without being logged through the operation and without a failure response attempt, so the caller only learns about it via invocation timeout. Please log it through the operation's logger and make sure the failure is surfaced.
4. **`catch (Throwable t)` swallowing `Error`s (F4).** Routing `OutOfMemoryError` and other `Error`s into the normal response path skips Hazelcast's `OutOfMemoryErrorDispatcher` hook that the generic-operation runner applied previously. Please narrow the catch to `Exception` or explicitly rethrow/dispatch `Error`s.
5. **Reflection on `Offload.executionService` in the test (F5).** `shouldAttemptSuccessResponseOnlyOnceWhenResponseDeliveryFails` reaches into Hazelcast's private field by name, which will break on upgrades. Once item 1 lands with an engine-owned executor, the test can inject that executor directly instead.
No new commit has appeared since `d71ee0a433fb`, so everything above will be re-checked once you push. Thanks for the work on this.
<!-- streview-comment:2018 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Offload report metrics operations [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11564:
URL: https://github.com/apache/seatunnel/pull/11564#issuecomment-5462271930
Follow-up with a more concrete next step than "just rerun."
I re-diagnosed the current failing `Build` run (`e01712f9`, same 4 jobs as before: `unit-test (11, windows-latest)`, `engine-v2-it (8, ubuntu-latest)`, `all-connectors-it-7 (11, ubuntu-latest)`, `doris-connector-it (8, ubuntu-latest)`) and looked specifically at `engine-v2-it`'s single failure: `SinkErrorToMysqlIT.testSinkMaxErrorRatioThreshold:615 » ConditionTimeout`.
That test class had a real flakiness fix land in `dev` the day after this PR's head was pushed: `73e46e0d627` / #11904 (`[Improve][E2E] Use DependencyJar for MySQL error tests`, 2026-08-20). It replaces a network `curl` download of the MySQL driver jar inside the test container with a local `DependencyJar.of(Driver.class).copyTo(...)` copy — removing exactly the kind of network-timing variance that produces `ConditionTimeout` failures in container-setup-sensitive E2E tests. This PR's head (`e01712f9`, 2026-08-19) predates that fix, and this branch is currently `behind_by=83` commits against `dev` (compare status: `diverged`).
So a plain "rerun the failed jobs" on the current head will very likely reproduce the same flake again, since the underlying test code hasn't changed. The lower-cost path is to sync this branch with the latest `dev` first (picking up #11904) and then rerun — none of the 4 failing jobs touch `ReportMetricsOperation`/`ReportMetricsOperationTest`, so I'd expect a clean run once the sync happens.
No new source-side finding from me — my 08-04 review conclusion still stands unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Offload report metrics operations [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11564:
URL: https://github.com/apache/seatunnel/pull/11564#issuecomment-5292327364
+1, thanks for the extra pair of eyes @nzw921rx -- agreed. I don't have any additional source-level blocker to add on this head either; the offload direction still looks correct and the remaining gate from my side is the normal write-capable maintainer review/merge step.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Offload report metrics operations [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11564:
URL: https://github.com/apache/seatunnel/pull/11564#issuecomment-5412507115
Confirmed, re-checked live just now: head is still `e01712f9`, `Build` is still `failure` on the same run (https://github.com/apache/seatunnel/runs/95934660228), nothing has changed since yesterday's check. Agreed this is the only remaining gate.
@nielifeng — the ball is in your court: a rerun of just the failed jobs (`gh run rerun <run-id> --failed` on the fork run behind this pointer check) on the current head `e01712f9`, then a note here once green. No new source-side finding from me; the APPROVED conclusion from 08-04 stands unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Offload report metrics operations [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11564:
URL: https://github.com/apache/seatunnel/pull/11564#issuecomment-6391360339
Thanks @SEZ9 for the thorough pass, and for confirming the head status. I agree with all five points and have nothing that contradicts them. My own review on 2026-10-23 for the same head `d71ee0a433fb` raised the same items, so I am +1'ing rather than duplicating them.
Two small additions so the author has the complete list in one place:
1. **Rejection of `executionService.execute(...)` (Low-Medium).** `Offload.start()` does not route a `RejectedExecutionException` (or any exception thrown by `execute(...)` itself) through `sendFailureResponse`, so the worker would wait for the invocation timeout. Wrapping the `execute(...)` call in `try/catch` and calling `sendFailureResponse(e)` closes that gap, and it pairs naturally with the dedicated/bounded executor discussion in your item 1.
2. **Suggested priority.** Items 1 and 2 (executor ownership/backlog and the Javadoc stating the real `Offload` contract) are the ones I would treat as must-fix before approval. The rest can ride in the same push.
On CI: the `Build` check on `d71ee0a433fb` currently shows `failure`. The failing fork jobs are `all-connectors-it-2`, `all-connectors-it-6`, `transform-v2-it-part-2`, `hbase-connector-it` and `mysql-cdc-connector-it` (JDK 11 and 17), none of which exercise `ReportMetricsOperation`, so they look unrelated to this diff. I have not triaged the individual logs, though, so once the code changes are pushed please re-run only the failed jobs on the new head and share the result.
@nielifeng, no new commit exists yet, so everything above will be re-checked once you push. If you disagree with any item (for example if you think the shared `OFFLOADABLE_EXECUTOR` is acceptable), a short note on the thread explaining the reasoning is perfectly fine. Thanks again for the work on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org