You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "DanielLeens (via GitHub)" <gi...@apache.org> on 2026/07/27 09:24:55 UTC
[PR] [Fix][Connector-V2] Propagate JDBC XA commit failures [seatunnel]
DanielLeens opened a new pull request, #11569:
URL: https://github.com/apache/seatunnel/pull/11569
### Purpose of this pull request
The JDBC XA commit path could let a checkpoint appear successful after a permanent resource-manager commit failure. `XaGroupOpsImpl` recorded the failure but did not throw it, while retryable XA exceptions were wrapped before the group retry logic could observe them. In the Zeta aggregated-commit path, returned attempt counters were also not persisted across checkpoint restart, so `max_commit_attempts` could not reliably bound retries.
This patch:
- propagates permanent XA commit failures to fail the checkpoint;
- preserves `XA_RETRY` and `XAER_RMFAIL` as retryable failures, while treating rollback codes such as `XA_RBTRANSIENT` as permanent commit failures;
- completes bounded aggregated-commit retries in one invocation so Zeta does not lose the incremented attempt state;
- ignores `XAER_NOTA` only when `restoreCommit` replays checkpoint state, keeping normal commits strict;
- adds focused regression tests for error classification, permanent-failure propagation, retry exhaustion, bounded aggregated retries, and normal-versus-restore commit behavior.
### Does this PR introduce _any_ user-facing change?
Yes. Previously, a permanent JDBC XA commit failure could be swallowed and the checkpoint could be reported as successful. Retryable failures could also bypass the intended retry path. After this change, transient no-effect/unknown-outcome failures are retried up to the configured limit, and permanent or exhausted failures fail the checkpoint instead of producing a false success. Recovery replay retains its existing idempotent handling for an already-completed transaction.
This closes the silent-failure gap; it does not make an absolute outcome claim for heuristic decisions or a non-compliant resource manager.
### How was this patch tested?
Added unit tests covering:
- `XA_RETRY` and `XAER_RMFAIL` transient classification;
- rollback and `XAER_NOTA` negative paths;
- permanent failure propagation and retry exhaustion;
- bounded retries within one aggregated-committer invocation;
- strict normal commit versus idempotent restore commit.
Local static validation passed:
```bash
./mvnw spotless:apply -pl seatunnel-connectors-v2/connector-jdbc -am -nsu -Dmaven.gitcommitid.skip=true
./mvnw spotless:check -pl seatunnel-connectors-v2/connector-jdbc -am -nsu -Dmaven.gitcommitid.skip=true
git diff --check
```
Compile, unit, and E2E tests were not run locally under the project review constraint; current-head GitHub CI is the functional validation gate.
### Check list
* [x] No new Jar binary package is added.
* [x] Documentation changes are not required because no configuration or user workflow changes.
* [x] No incompatible configuration, public SPI, or serialized-state change is introduced.
* [x] Connector registration, plugin mapping, distribution POM, plugin config, and E2E registration changes are not applicable to this existing connector bug fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Propagate JDBC XA commit failures [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11569:
URL: https://github.com/apache/seatunnel/pull/11569#issuecomment-5554206317
CI status (maintainer triage): both previous Builds (b8d2dbad1, ba3c92216) failed `updated-modules-integration-test-part-2` on `XaGroupOpsImplIT` with `NotFoundException: Status 404: No such image: mysql:8.0.43`. That is deterministic, not a registry blip: `AbstractJdbcIT#tearDown` removes the database image from the Docker daemon after every JDBC IT, `JdbcMysqlIT` uses the same `mysql:8.0.43` image and runs in the same surefire JVM right before `XaGroupOpsImplIT`, and Testcontainers caches "image is present" per JVM, so the default pull policy skipped the pull and `createContainer` failed. Pushed ce28494b, which sets `PullPolicy.alwaysPull()` on this test's `MySQLContainer` so the image is re-fetched whenever a previous test removed it. The other failures on ba3c92216 were unrelated: `updated-modules-integration-test-part-2 (8)` was a Maven Central read timeout resolving `org.testcontainers:postgresql:1.17.6`, and `doris-connector-it (8)` was `DorisErrorIT` (#12114).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Propagate JDBC XA commit failures [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11569:
URL: https://github.com/apache/seatunnel/pull/11569#issuecomment-5340775131
Thanks for the ping, @davidzollo, and for re-checking the restore reconciliation.
This is a re-verification of the current head, `17234b67e1d5574f116358f7b8f3b74a862ce496`. I want to be precise about what actually changed since my last review, `03f7fd4347f7870d494b8e2babb2f362e947ec74` (2026-08-19, a few hours ago), because it matters for what this comment is and isn't:
```
git diff 03f7fd4347f7..17234b67e1d5 --stat
```
returns **empty**. The only commit on top of `03f7fd4347f7` is `17234b67e1d` itself, `[Chore] Retrigger CI for PR #11569` — an empty commit with no file changes. So this is not a new code version; it is the exact same diff I already did a full from-scratch re-trace against this morning. I'm not going to pretend to "discover" things again on identical code — I'll restate my open conclusion precisely and respond to the new activity since then.
# What Problem Does This PR Solve?
- User pain point: in the JDBC XA (2PC) aggregated-commit path, a permanent XA commit failure could be silently swallowed (`throwIfAnyFailed("commit")` was commented out, and `TransientXaException` was double-wrapped so its catch clause was unreachable dead code). A checkpoint could be reported successful while the transaction stayed prepared-but-uncommitted in the database.
- Fix approach: reclassify `XA_RBTRANSIENT` -> `XA_RETRY`, make `wrapException` return instead of throw, re-enable `throwIfAnyFailed`, and reconcile checkpoint-restored XIDs against a fresh `xaFacade.recover()` scan using a commit-order (prefix/suffix) invariant rather than a blanket `XAER_NOTA` tolerance.
- One-sentence summary: this turns a silent "reports success, loses the commit" bug into a fail-closed commit/restore path — but the current fail-closed branch for "no evidence either way" is itself unsound, which is Issue 1 below.
# 1. Code Change Review
## 1.1 Core Logic Analysis — status of my own Issue 1
Re-read the code in place on the current head to confirm nothing moved:
`JdbcSinkAggregatedCommitter.java:193-215` (`replayRecoveredCheckpoint`):
```java
int firstRecoveredIndex = findFirstRecoveredIndex(checkpointXids, recoveredXids);
if (firstRecoveredIndex < 0) {
throw new JdbcConnectorException(
CommonErrorCodeDeprecated.WRITER_OPERATION_FAILED,
String.format(
"none of the restored checkpoint transactions are present in the XA recovery scan: %s",
checkpointXids));
}
```
Unchanged from what I traced this morning. `testRestoreCommitFailsClosedWhenRecoveryScanHasNoCheckpointXid` (`JdbcSinkAggregatedCommitterTest.java:144-164`) still asserts exactly this throw, so the behavior is intentional and still covered — it's just the wrong outcome for the case it covers.
The problem, restated once more for the record since it's the actual blocker: `CheckpointCoordinator` sets `latestCompletedCheckpoint` as the durable restore point *before* the phase-2 XA commit for that checkpoint even runs (`CheckpointCoordinator.java:1330` precedes the `notifyCompleted()` call at `:1335`), and it stays pointed at that checkpoint until the *next* checkpoint independently completes. Any restart in that window — for a reason having nothing to do with XA at all (an unrelated source/task failure, a node loss) — replays a `restoreCommit()` call whose XidInfo list has, by then, already been fully and successfully committed by a prior, healthy `notifyCheckpointComplete()`. Every XID in that batch is legitimately absent from the resource manager (successfully committed and forgotten, standard XA behavior), so `findFirstRecoveredIndex` returns `-1` for the *entire* batch — and the code now treats total absence as proof of an unrecoverable problem rather than the si
ngle most likely, most benign explanation. Because the RM state driving this decision doesn't change between restart attempts, the job is stuck permanently until an operator manually intervenes.
This is a real regression relative to every earlier version of this PR I reviewed, including the one @nzw921rx and @davidzollo raised CHANGES_REQUESTED against: the pre-`dd143c0445c6` code tolerated every absent XID unconditionally, which correctly handled the "whole batch already committed" case (at the cost of the narrower data-loss risk both of you originally, and correctly, flagged). The current code closes that data-loss risk but, in doing so, converts the single most common healthy-restart case into a hard failure loop. That's a worse outcome for availability than the original bug was for correctness, and it doesn't require reverting the good parts of this rework — see the fix direction in Issue 1 below.
## 1.2 Compatibility Impact
**Classification: Fully compatible at the API/config/serialization level**, unchanged from my prior assessment — no public API break, no config key/default change, no checkpoint state schema change (`XidInfo`/`JdbcAggregatedCommitInfo` untouched). The intended behavioral tightening (restore can now fail rather than silently guess) is documented in `incompatible-changes.md` in both `docs/en` and `docs/zh`. That documentation is accurate for the code as written, but the code it describes has the Issue 1 bug, so the compatibility note currently undersells the operational risk for the all-absent case specifically.
## 1.3 Performance / Side-Effect Analysis
Unchanged from this morning: `recoverCheckpointTransactions()` adds one bounded-retry `xaFacade.recover()` scan on restore only (not steady-state commit); `commitXidInfos`'s round cap has no backoff between rounds (Issue 2, carryover, still open) — `XAER_RMFAIL`-class outages will still burn the whole `max_commit_attempts` budget in microseconds.
## 1.4 Error Handling and Logging
Same two open items as this morning's review, both carryovers, neither new:
- **Issue 1 (Blocking):** the `firstRecoveredIndex < 0` branch (above).
- **Issue 2 (Medium):** no backoff between synchronous retry rounds (`JdbcSinkAggregatedCommitter.java:124-142`, `:234-255`).
# 2. Code Quality Assessment
## 2.1 Coding Standards
Unchanged: Javadoc on all new/changed methods matches their bodies, ASF headers present, no wildcard imports, no `System.out.println`, AOSP formatting consistent with the module.
## 2.2 Test Coverage and Test Stability
Test content is byte-for-byte identical to what I reviewed this morning (confirmed via the empty `git diff --stat`). **Stability rating: Stable** — all tests remain Mockito-based, single-threaded, deterministic on object identity/counts/exception message content; no `Thread.sleep`, no shared static state, no `@DisplayName` usage.
**Gap (Issue 3, Medium, carryover, still open):** there is still no test where the *entire* checkpoint batch is legitimately, successfully already-committed (a prior healthy `notifyCheckpointComplete()`, followed by an unrelated restart) — as opposed to a prior *aborted* restore attempt. That's the exact scenario Issue 1 describes, and it should be the first regression test added alongside the fix. `XaGroupOpsImplIT` remains `@Disabled`; there is still no real-database exercise of any part of this PR's XA commit/restore path.
## 2.3 Documentation Updates
`docs/en/connectors/sink/Jdbc.md`, `docs/zh/connectors/sink/Jdbc.md`, and both `incompatible-changes.md` files document the restore-order/fail-closed behavior consistently in both languages. As in 1.2, this is accurate documentation of code that currently has the Issue 1 bug, so it should be revisited once that's fixed rather than treated as a separate doc task.
# 3. Architectural Soundness
## 3.1 Elegance of the Solution
The commit-order/prefix-suffix reconciliation is a genuinely better idea than either the original strict-everywhere or tolerant-everywhere positions from the two earlier CHANGES_REQUESTED reviews — it grounds the tolerance decision in a real invariant of this code's own sequential commit loop instead of an unprovable assumption about resource-manager history. The remaining problem is narrow: the boundary condition where the model has *zero* still-prepared evidence to anchor the prefix/suffix split against. Fixing that doesn't require touching the rest of the design.
## 3.2 Maintainability
Good: the responsibility split (`XaFacadeImplAutoLoad` classifies driver outcomes, `XaGroupOpsImpl` owns grouped propagation, `JdbcSinkAggregatedCommitter` owns restore reconciliation) is clean, and the helper methods (`replayRecoveredCheckpoint`, `findFirstRecoveredIndex`, `recoverCheckpointTransactions`) are small and single-purpose.
## 3.3 Extensibility
Unchanged: `XidKey` is a solid reusable canonical-comparison primitive; the `default` method on `XaGroupOps.commit(...)` lets other implementors opt in incrementally without a forced signature change.
## 3.4 Historical-Version Compatibility
No checkpoint-state schema change; old checkpoints/savepoints deserialize unmodified under this code. Issue 1 is a decision-logic problem, not a serialization/format compatibility problem, and it affects every upgrade path equally regardless of checkpoint age.
# 4. Issue Summary
| # | Issue | Location | Severity | Status |
|---|-------|----------|----------|--------|
| 1 | All-absent restore batch throws unconditionally, misclassifying "already fully committed by a prior healthy checkpoint, unrelated restart" (the common case) as unrecoverable data loss; recurs on every restart since the underlying RM state never changes, wedging the job permanently | `JdbcSinkAggregatedCommitter.java:198-205` | High (Blocking) | Open, carried forward unchanged from my 2026-08-19T04:31 review; code is byte-identical |
| 2 | No backoff between synchronous commit/recover retry rounds; `max_commit_attempts` is largely inert against `XAER_RMFAIL`-class outages | `JdbcSinkAggregatedCommitter.java:124-142`, `:234-255` | Medium | Open, carryover from 2026-08-16 |
| 3 | No end-to-end or real-database test exercises commit/restore failure propagating through to checkpoint/job failure, and specifically nothing exercises "whole batch already committed by a prior successful commit"; `XaGroupOpsImplIT` remains `@Disabled` | test classes; `XaGroupOpsImplIT.java` | Medium | Open, carryover from 2026-08-16 |
| 4 | CI on the current head (`17234b67e1d`) is a plain retrigger with no functional diff; the fork's `Build` run for this exact SHA is still `queued` as of this comment. The prior run against identical code (`03f7fd4347f7`, fork run `32162717118`) was green on every JDBC/XA-relevant job (unit-test, updated-modules-integration-test parts 1-8); its only failure was an unrelated Paimon E2E flake | fork run `32235473661` (queued); prior identical-code run `32162717118` | Low (process, not source) | Not yet resolved (run pending) |
# 5. Merge Recommendation
### Conclusion: Not recommended for merge
1. **Blockers — must be fixed**
- **Issue 1 (High):** the all-absent branch in `replayRecoveredCheckpoint` must stop treating "no evidence either way" as proof of failure. Concretely: `firstRecoveredIndex < 0` should log at WARN and return (treating the whole batch as already resolved), not throw — this is the single most likely, most benign explanation for total absence, unlike the partial-absence "gap after a still-prepared XID" branch at `:209-215`, which is correctly kept as fail-closed because that pattern cannot arise from a legitimate sequential commit. This is a one-branch fix plus one new regression test ("all-XIDs-already-committed restore batch completes without error"), not a redesign — the surrounding commit-order/prefix-suffix machinery is sound and should be kept exactly as-is.
2. **Recommended fixes — non-blocking**
- Issue 2 (Medium): add bounded backoff between synchronous retry rounds.
- Issue 3 (Medium): add a committer-level test exercising the "whole batch already committed" restore scenario, and re-enable `XaGroupOpsImplIT` with real XA coverage.
- Issue 4 (Low): let the pending fork CI run for `17234b67e1d` complete; no action expected beyond confirming it matches the already-green result for the identical `03f7fd4347f7` code.
**On @davidzollo's request to re-check and clear the stale review gate:** the restore reconciliation you asked reviewers to re-check does correctly close both your 2026-08-06 idempotency finding and @nzw921rx's 2026-07-27 normalization finding — I independently re-verified that again on this pass (see 3.1). But it isn't ready to clear yet, because I found a new, more severe issue in the same reconciliation logic on 2026-08-19 (Issue 1 above), a few hours before your comment. That one hasn't been addressed by any commit since — the only thing that landed after your comment was the CI-retrigger no-op. So `reviewDecision=CHANGES_REQUESTED` should stay in place, now anchored to Issue 1 rather than to either of the original two findings.
**Overall assessment:** the underlying engineering direction is sound and has meaningfully improved across every round of review on this PR. What's blocking merge right now is one specific branch's failure-mode choice, not the architecture around it. Once `firstRecoveredIndex < 0` returns instead of throwing and that's backed by a regression test, I would expect to be able to approve this on the next pass.
Since this is my own PR, this is posted as a comment rather than an approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Propagate JDBC XA commit failures [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11569:
URL: https://github.com/apache/seatunnel/pull/11569#issuecomment-5473674263
Note: this PR is authored by me (DanielLeens), so GitHub blocks a self-review submission. Posting this as a plain issue comment instead of a formal review, per project convention for self-authored PRs (same as my previous rounds on this thread).
New activity since my last comment (`d2bc272f3641`, 2026-08-30T01:50:21Z): four real commits plus one empty CI-retrigger commit, current head `5eefa194acda`:
- `7a951674dafb` "[Fix][E2E] Stabilize CDC and Doris error tests" - touches `PostgresCDCIT.java`, `DorisErrorIT.java`
- `31cf042b905a` "[Fix][E2E] Extend HANA startup readiness timeout" - touches `JdbcHanaIT.java`
- `091f58e37f00` "[Fix][Zeta] Stabilize pending job cleanup regression test" - touches `CoordinatorServiceTest.java`
- `1a7a9dd2626d` "[Fix][E2E] Stabilize Postgres CDC restore test timing" - touches `PostgresCDCIT.java`, `CoordinatorServiceTest.java`
- `5eefa194acda` "[Chore] Refresh CI" (current head) - verified via `gh api repos/apache/seatunnel/commits/5eefa194acda --jq '.files | length'` -> `0`, a genuine empty CI retrigger.
I diffed `d2bc272f3641...5eefa194acda` directly rather than trusting the commit subjects: the only four changed files across this round are `PostgresCDCIT.java`, `DorisErrorIT.java`, `JdbcHanaIT.java`, and `CoordinatorServiceTest.java`. None of `JdbcSinkAggregatedCommitter.java`, `XaGroupOpsImpl.java`, `XaFacadeImplAutoLoad.java`, or `GroupXaOperationResult.java` - the files this PR's actual fix lives in - changed at all this round. So this is not a from-scratch re-review; the core-logic conclusion from my 2026-08-30 comment stands unchanged.
# What this round actually is
All four substantive commits are CI-stability fixes for test flakes that earlier rounds on this exact PR had already identified as unrelated pre-existing flakiness (the Postgres CDC restore-timing race, the HANA five-minute tenant-DB startup window, and the coordinator pending-job-scheduler test's dependence on a synchronous `await().untilAsserted(Mockito.verify(...))` poll instead of a deterministic latch), not new work on the XA commit-failure fix itself. I spot-checked each patch instead of assuming the message matches the diff:
- `PostgresCDCIT` (`7a951674dafb`, `1a7a9dd2626d`): moves the row insert for the restored job to after `waitForReplicationSlotActive(...)` and adds an explicit wait for the old replication slot to go inactive before treating the restored one as the current state. This removes a real race (inserting before the restored slot could consume it) rather than loosening any assertion - the `await().untilAsserted(...)` checks are unchanged in strictness, only the ordering/timing around them changed.
- `JdbcHanaIT` (`31cf042b905a`): raises the container startup timeout from 5 to 10 minutes with a comment explaining HANA creates its tenant database after the process starts. A timeout increase for a genuinely slow, non-deterministic external dependency; the assertion being waited on (`Startup finished!` log line) is unchanged.
- `CoordinatorServiceTest` (`091f58e37f00`, `1a7a9dd2626d`): replaces a `Mockito.verify(..., atLeastOnce())` polled via `await()` with a `CountDownLatch` that the mocked `preApplyResources()` call counts down itself, and runs the scheduler on the test's own executor (then, in the second commit, swaps in the production `executorService` field directly) instead of a detached thread. This is a strictly more deterministic synchronization primitive replacing a race-prone poll, not a weaker check - the final assertions (`pendingJobQueue` no longer contains the job, `interrupt()` was called) are byte-for-byte the same as before.
- `DorisErrorIT` (`7a951674dafb`): the one change worth flagging on its own merits even though it's out of this PR's scope - it swaps an assertion on a specific stack-trace substring (`...RecordBuffer.checkErrorMessageByStreamLoad`) for one on `DorisConnectorErrorCode.STREAM_LOAD_FAILED.getDescription()`, while keeping the existing `getCode()` assertion and the non-zero exit code check untouched. That's trading a brittle white-box check (an internal method name that any unrelated refactor could rename) for a still-specific check on the connector's own public error-code description - I don't read this as a coverage reduction, but it's tangential to this PR and outside what I fixed here, so I'm noting it rather than owning it as part of this round's review.
None of this round's commits touch JDBC, XA, or this PR's checkpoint/restore reconciliation logic, and none of it weakens an assertion, timeout direction, or coverage scope - it tightens timing determinism in every case I checked.
# CI
Dereferenced the apache `Build` pointer on the current head (`5eefa194acda`) to the real fork run: `DanielLeens/seatunnel` run `33355857983`. Unlike the last two rounds where I flagged failing job buckets before logs were retrievable, this run is progressing cleanly so far, about 15 minutes in: `changes`, `Sanity check results`, `License header`, `Dead links`, `Code style`, and `Check Helm Chart` have all completed successfully; the large integration-test matrix (`all-connectors-it-1` through `-8`, `engine-v2-it`, `transform-v2-it-part-1/2`, `unit-test` on both JDKs/OSes) is `in_progress`; `jdbc-connectors-it-part-1` and `all-connectors-it-2`/`-3` are still `queued` waiting for a runner slot. Zero failures on this head so far - a meaningfully cleaner start than the three early failures I saw and couldn't yet diagnose on the previous head (`d2bc272f3641`).
# Process status (unchanged from my last comment)
`reviewDecision` is still `CHANGES_REQUESTED`, still driven solely by @nzw921rx's 2026-07-27 review, which - as I established in my 2026-08-29 full re-review - predates the recovery-scan reconciliation mechanism entirely and targets a design that no longer exists in this form. That review has not been revisited, updated, or dismissed since I last flagged it, and I cannot do so myself as the PR author. `mergeable_state` is still `blocked` for the same reason. The two low-severity carryovers from @li3zhi4 (PR description wording on the all-absent case, the still-unused `ignoreUnknown=true` path) also remain open and unaddressed this round.
# Conclusion
No change to my 2026-08-30 merge recommendation: **Ready to merge** on source correctness once CI finishes green. The only blockers are process, not code: @nzw921rx's stale `CHANGES_REQUESTED` needs a maintainer with write access to revisit or dismiss it, and this run's CI needs to finish (it's healthy so far, no need to intervene).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Propagate JDBC XA commit failures [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11569:
URL: https://github.com/apache/seatunnel/pull/11569#issuecomment-5396756323
CI failure triage for the last red Build (fork run 32582046522, both attempts, head `0d9f281e054a`):
1. **`kafka-connector-it` — deterministic on the stale branch, not caused by this PR.** Both attempts failed on the identical test, line and container: `KafkaIT.testKafkaToKafkaExactlyOnceOnStreaming{TestContainer}[2]` timing out in `awaitStreamingPipelineReady` (KafkaIT.java:1327), on java 8 and java 11. The branch was based on 2026-08-13 dev (95 commits behind) and misses the Kafka exactly-once fixes merged since: #11541 (exactly-once sink losing the first record on checkpoint), #11231 (wait for exactly-once test sends), plus #11572/#10925 e2e stabilizations. This PR only touches the JDBC XA path, so the exactly-once Kafka failure is stale-branch fallout.
2. **`unit-test (11, windows-latest)` — known environmental flake**: `IllegalStateException: Node failed to start!` in `seatunnel-engine-server` (`CheckpointPlanTest`, 311s). The same Hazelcast node-start flake has hit unrelated PRs on windows-latest with a rotating test class. The java-8 twins were fail-fast matrix cancellations of the same event; the branch also predates #11873, which set `fail-fast: false`.
**Action taken:** merged current dev (`43fe63b1fca`) into the branch as a maintainer edit — no conflicts; the diff vs dev is still exactly the PR's XA changes plus docs. `spotless:apply -pl seatunnel-connectors-v2/connector-jdbc` clean. A fresh Build run is in progress on head `403a0b09f88d`. If the Windows Hazelcast flake strikes again it only needs a job-level rerun, not code changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Propagate JDBC XA commit failures [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on code in PR #11569:
URL: https://github.com/apache/seatunnel/pull/11569#discussion_r3886927168
##########
seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/sink/JdbcSinkAggregatedCommitter.java:
##########
@@ -70,22 +80,76 @@ private void tryOpen() throws IOException {
@Override
public List<JdbcAggregatedCommitInfo> commit(
List<JdbcAggregatedCommitInfo> aggregatedCommitInfos) throws IOException {
+ return commitPreparedTransactions(aggregatedCommitInfos);
+ }
+
+ /**
+ * Reconciles checkpoint XIDs with the resource manager using commit-order evidence. Checkpoint
+ * XIDs from the first still-prepared transaction onward must all be present in the recovery
+ * scan and are replayed strictly. An all-absent batch is treated as already resolved, while an
+ * absent prefix before a still-prepared suffix is treated as already resolved only after that
+ * suffix commits successfully.
+ */
+ @Override
+ public List<JdbcAggregatedCommitInfo> restoreCommit(
+ List<JdbcAggregatedCommitInfo> aggregatedCommitInfos) throws IOException {
+ tryOpen();
+ for (JdbcAggregatedCommitInfo aggregatedCommitInfo : aggregatedCommitInfos) {
+ // Refresh RM evidence for every batch because transactions may be resolved concurrently
+ // during failover while earlier restored batches are being replayed.
+ replayRecoveredCheckpoint(
+ aggregatedCommitInfo.getXidInfoList(), recoverCheckpointTransactions());
+ }
Review Comment:
Addressed in c5f4ec0a0d. The Javadoc now identifies DBA or resource-manager cleanup as the external actor, explains that no second Zeta committer races the restore, and states that the per-batch recovery scan narrows rather than eliminates the gap before XA commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Propagate JDBC XA commit failures [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11569:
URL: https://github.com/apache/seatunnel/pull/11569#issuecomment-5488644802
# What Problem Does This PR Solve?
Before this PR, a permanent JDBC XA commit failure could be silently swallowed instead of
propagated: `GroupXaOperationResult.throwIfAnyFailed("commit")` was commented out, and
`XaFacadeImplAutoLoad.wrapException()` always pre-wrapped `TransientXaException` inside a
`JdbcConnectorException`, so `XaGroupOpsImpl.commit()`'s `catch (XaFacade.TransientXaException e)`
branch was dead code. A prepared XA transaction could fail to commit and vanish without being
retried, reported, or rolled back, while the checkpoint was still reported complete — a real
data-loss risk for a two-phase-commit sink. The fix restores failure propagation, corrects
transient-vs-permanent XA error classification, adds bounded synchronous retry with backoff, and
adds a `restoreCommit()` reconciliation path that consults a live `xaFacade.recover()` scan in
commit order before replaying restored, still-in-doubt transactions.
# Re-review context (why this round happened)
This is a follow-up check, not a new code round. The head SHA is unchanged since my last full
review: `18bfbd1743bde7b9ab270545665139f5ad0f72f6`. I confirmed this directly (`git log -1` on a
fresh fetch of `refs/pull/11569/head`, and `gh pr view --json headRefOid`), and re-read
`JdbcSinkAggregatedCommitter.java` at that exact commit to double check nothing drifted between my
last review and this one — it hasn't. My prior review already did the full source-level trace of
`XaFacadeImplAutoLoad`, `XaGroupOpsImpl`, `GroupXaOperationResult`, and
`JdbcSinkAggregatedCommitter.restoreCommit()`/`replayRecoveredCheckpoint()`, and I stand by that
analysis unchanged (see 1.1 below for a condensed restatement rather than a full re-derivation).
What's actually new in this round is CI progress: at my last review the fork's `Build` run for this
exact head had just been queued (0 jobs finished). It has now run substantially further.
# 1. Code Change Review
## 1.1 Core Logic Analysis (condensed restatement, unchanged from my last pass)
- `XaFacadeImplAutoLoad.TRANSIENT_ERR_CODES` correctly changed from `{XA_RBTRANSIENT, XAER_RMFAIL}`
to `{XA_RETRY, XAER_RMFAIL}`. `XA_RBTRANSIENT` means the branch is already rolled back (a done,
permanent outcome); `XA_RETRY` means the call had no effect and may be safely reissued. The old
code retried an already-rolled-back branch, which could never succeed.
- `wrapException()` now returns `TransientXaException` bare instead of nesting it inside
`JdbcConnectorException`, which is what makes `XaGroupOpsImpl.commit()`'s
`catch (XaFacade.TransientXaException e)` reachable for the first time. All three call sites
still `throw wrapException(...)` — verified.
- `restoreCommit()`/`replayRecoveredCheckpoint()` reconciles checkpoint XIDs against a fresh
recovery scan using commit order: everything before the first still-prepared XID is treated as
already resolved, the still-prepared suffix is replayed strictly, and a gap after the first
still-prepared XID fails closed (this relies on `XaGroupOpsImpl.commit()` only ever attempting
XIDs in original order with `allowOutOfOrderCommits=false`, so a still-prepared XID at index *k*
implies everything after *k* in the same batch must also still be prepared — a genuine mismatch
there is real evidence of inconsistency, not something to guess past).
**Runtime path reached: yes, this is the normal restore path**, not a rare corner case —
`restoreCommit()` runs on every task restart with in-flight XA checkpoint state.
## 1.2 Compatibility Impact
Unchanged from my last review: partially incompatible, and disclosed. No wire/state serialization
format change. The observable behavior change — a permanent XA commit failure now fails the
checkpoint instead of being silently absorbed — is the intended fix and is documented in
`docs/en(zh)/introduction/concepts/incompatible-changes.md`.
## 1.3 Performance / Side-Effect Analysis
Unchanged: bounded retry (`maxCommitAttempts` default 3, 1s backoff), runs on the aggregated
committer's own dedicated task thread, not a shared engine-wide thread. No unbounded structures.
## 1.4 Error Handling and Logging
No new issues found this round. Same as before: every hard-failure path throws a
`JdbcConnectorException` naming the affected XIDs, and logging levels are appropriate.
# 2. Code Quality Assessment
## 2.1 Coding Standards
Unchanged from my last review — Javadoc present on all new/changed non-trivial methods, ASF
headers present, no wildcard imports.
## 2.2 Test Coverage and Test Stability
**Core JDBC/XA logic: Stable**, unchanged assessment. What's new this round is that the CI jobs
that actually exercise this coverage have now run:
- `jdbc-connectors-it-part-1` through `jdbc-connectors-it-part-7`, both JDK 8 and JDK 11 (14 jobs
total): **all green.** `jdbc-connectors-it-part-1` carries the re-enabled `XaGroupOpsImplIT`
(previously `@Disabled` for a MySQL-driver classloading reason), which this PR's own real-MySQL
test of failure propagation through the aggregated committer depends on. This directly closes the
Low-severity "verify via CI" caveat I flagged as Issue 1 in my prior review — the re-enable is
now proven, not just plausible.
- `unit-test`, both JDK 8 and JDK 11: **all green.** This carries
`XaFacadeImplAutoLoadTest`/`XaGroupOpsImplTest`/`JdbcSinkAggregatedCommitterTest` and the
`CoordinatorServiceTest` change I flagged as Issue 2 (four consecutive commits stabilizing the
same test). Green on this run closes that verification request too.
**Stability rating: Stable** (unchanged from last review — no `Thread.sleep`, no shared static
state, no order-dependence, no floating-point comparisons in any of the touched test code).
## 2.3 Documentation Updates
Unchanged — already covered by earlier rounds and not affected by this round.
# 3. Architectural Soundness
Unchanged from my last review — precise fix, good maintainability and extensibility, no checkpoint
state schema change (see prior review for full detail; nothing in this round's diff touches these
dimensions since there is no new diff).
# 4. Issue Summary
| # | Issue | Location | Severity | Status |
| --- | --- | --- | --- | --- |
| 1 | `XaGroupOpsImplIT` re-enable wasn't proven by the diff alone | `.../connector-jdbc-e2e-part-1/.../XaGroupOpsImplIT.java` | Low | **Resolved this round** — `jdbc-connectors-it-part-1` is green on both JDK 8 and 11 in fork run `33453354584` |
| 2 | `CoordinatorServiceTest` re-touched four rounds in a row for the same race | `.../engine/server/CoordinatorServiceTest.java` | Low | **Resolved this round** — `unit-test` is green on both JDK 8 and 11 in the same run |
| 3 | Two unrelated CI job failures in the broader `all-connectors-it` matrix | see CI diagnosis below | Informational, not a PR blocker | New this round, diagnosed below |
| 4 (carried) | `XaFacade.commit(xid, ignoreUnknown=true)` still has no production caller | `JdbcSinkAggregatedCommitter.java` / `XaFacadeImplAutoLoad.java` | Low | Open, unchanged |
| 5 (carried) | PR description still describes the all-absent case as "fails closed" when code/docs treat it as already-resolved | PR description text | Low | Open, unchanged |
No High or Medium issues found this round or carried from before. Issues 1 and 2 from my prior
review are now closed by the CI evidence below; Issue 3 is a fresh CI observation, diagnosed as
unrelated to this PR's diff.
## CI diagnosis (task-relevant detail)
Fork `Build` run for the current head: `https://github.com/DanielLeens/seatunnel/actions/runs/33453354584`
(the apache-side "Build" check is a pointer to this run, per this repo's usual CI wiring).
- 86 of 93 jobs completed, 7 still `in_progress` at the time of this comment
(`engine-v2-it`, `mysql-cdc-connector-it` x2 JDK, `connector-redis-it` x2 JDK,
`kafka-connector-it` x2 JDK — none of these touch the JDBC XA code path).
- 84 of the 86 completed jobs are green, including every `jdbc-connectors-it-part-*` (1-7, both
JDKs) and every `unit-test` (both JDKs) — the jobs that actually exercise this PR's own diff.
- 2 failures, both clearly unrelated to this PR:
- `all-connectors-it-4 (8, ubuntu-latest)`:
`org.apache.seatunnel.e2e.connector.azurecosmosdb.AzureCosmosDBSourceIT` — an Azure Cosmos DB
source test, a connector this PR does not touch.
- `all-connectors-it-7 (11, ubuntu-latest)`:
`org.apache.seatunnel.e2e.connector.v2.milvus.MilvusIT` — a Milvus test, also a connector this
PR does not touch.
- The rest of the `all-connectors-it-*`/`updated-modules-integration-test-part-*` split shows the
usual round-robin-bucket pattern for this repo's CI (some `updated-modules-*` legs `skipped`
because the broader `all-connectors-it-*` legs ran instead) — this is expected bucket routing,
not a coverage gap.
- Conclusion: nothing in the current CI signal implicates this PR's JDBC/XA change. The two
failures are pre-existing/unrelated-connector flakes in a shared matrix job, and every job that
actually covers this diff is green.
# 5. Merge Recommendation
### Conclusion: Ready to merge after fixes
This round found no new code-correctness issues — there is no new code to find them in, the diff is
byte-identical to what I already fully reviewed. The two Low-severity CI-verification caveats from
my prior review are now closed with real, observed green results, not just plausibility arguments.
1. **Blockers — must be fixed (process, not code; I cannot act on these myself as the PR author):**
- `reviewDecision` is still `CHANGES_REQUESTED`, driven by two still-open, unresolved formal
reviews: `@nzw921rx` (2026-07-27) and `@davidzollo` (2026-08-06). Both predate the current
recovery-scan/commit-order reconciliation design and, based on my own independent source-level
re-derivation in earlier rounds of this review, are addressed by the code as it stands today.
Neither reviewer has formally dismissed or updated their review state on GitHub. This needs a
maintainer with write access to re-review and clear the stale gate, or `@nzw921rx` /
`@davidzollo` to do so themselves — I cannot dismiss another reviewer's formal review.
- Let the remaining 7 in-progress CI jobs finish. None of them touch this PR's own code path, but
a fully green run (rather than a run with jobs still pending) should be confirmed before merge.
2. **Recommended fixes — non-blocking, unchanged from before:**
- Carried Issue 4 (Low): remove or wire up the unused `ignoreUnknown=true` production path.
- Carried Issue 5 (Low): tighten the PR description's wording on the all-absent-batch behavior to
match the code/docs ("treated as already resolved," not "fails closed").
**Overall assessment:** the core fix is sound and has converged, not regressed, across every round
of review on this PR — I have no new source-level concerns this round because there is no new
source in this round. The two open blockers are both procedural: a stale, unresolved
`CHANGES_REQUESTED` state from reviews that predate the current design, and a CI run that hasn't
finished yet (but is green everywhere relevant to this diff so far). Once a maintainer clears the
stale review gate and the remaining CI jobs finish green, I see nothing left blocking merge.
Since this is my own PR, this is posted as a comment rather than an approval/changes-request —
GitHub blocks self-review states. A different maintainer needs to action the review-gate item above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Propagate JDBC XA commit failures [seatunnel]
Posted by "dybyte (via GitHub)" <gi...@apache.org>.
dybyte commented on code in PR #11569:
URL: https://github.com/apache/seatunnel/pull/11569#discussion_r3870383845
##########
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-part-1/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/xa/XaGroupOpsImplIT.java:
##########
Review Comment:
Would it be worth adding a non-blocking E2E test covering partial XA commit followed by Zeta restart and recovery? The unit tests cover the reconciliation logic well, but this would validate the full checkpoint → failure → restore path.
##########
seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/sink/JdbcSinkAggregatedCommitter.java:
##########
@@ -70,22 +80,76 @@ private void tryOpen() throws IOException {
@Override
public List<JdbcAggregatedCommitInfo> commit(
List<JdbcAggregatedCommitInfo> aggregatedCommitInfos) throws IOException {
+ return commitPreparedTransactions(aggregatedCommitInfos);
+ }
+
+ /**
+ * Reconciles checkpoint XIDs with the resource manager using commit-order evidence. Checkpoint
+ * XIDs from the first still-prepared transaction onward must all be present in the recovery
+ * scan and are replayed strictly. An all-absent batch is treated as already resolved, while an
+ * absent prefix before a still-prepared suffix is treated as already resolved only after that
+ * suffix commits successfully.
+ */
+ @Override
+ public List<JdbcAggregatedCommitInfo> restoreCommit(
+ List<JdbcAggregatedCommitInfo> aggregatedCommitInfos) throws IOException {
+ tryOpen();
+ for (JdbcAggregatedCommitInfo aggregatedCommitInfo : aggregatedCommitInfos) {
+ // Refresh RM evidence for every batch because transactions may be resolved concurrently
+ // during failover while earlier restored batches are being replayed.
+ replayRecoveredCheckpoint(
+ aggregatedCommitInfo.getXidInfoList(), recoverCheckpointTransactions());
+ }
Review Comment:
Could you clarify which actor can concurrently resolve these checkpoint XIDs during failover? From the Zeta lifecycle, I see one aggregated committer task per sink, pipeline restore happens after the old task group reaches a terminal state, and master failover explicitly skips redeployment when the same TaskGroupLocation is still active. If the concurrent resolver is external to Zeta, a fresh recovery scan per batch only reduces stale evidence; it does not prevent a transaction from being resolved between recover() and commit(). Is per-batch recovery actually required for correctness here, or is it only a defensive freshness measure?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Propagate JDBC XA commit failures [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11569:
URL: https://github.com/apache/seatunnel/pull/11569#issuecomment-5509573016
@/private/tmp/claude-501/-Volumes-disk6-tmp-seatunnel-third-lucid-dewdney-453831/0f18693c-e86b-49b7-acfa-1816b0b91004/scratchpad/pr_comment.txt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Propagate JDBC XA commit failures [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11569:
URL: https://github.com/apache/seatunnel/pull/11569#issuecomment-5564232164
CI status (maintainer triage): the Build on `b56f423a1` was `cancelled` — `updated-modules-integration-test-part-6 (11)` hit its 210-minute job timeout. Traced the log: `FlussSinkIT` (module `connector-fluss-e2e`, bucketed into this run alongside `connector-jdbc-e2e-part-5` and `connector-cdc-mongodb-e2e`) started at 11:19 and never completed — the Zeta engine coordinator looped its idle thread-pool/job-status diagnostic dump every 10s from 11:54 until the 14:45 timeout kill, with no further test progress in between. Unrelated to this PR's diff, which only touches `connector-jdbc`'s XA path and `XaGroupOpsImplIT` (part-1, not part-5). Retriggered just the timed-out job (not a full rebuild).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Propagate JDBC XA commit failures [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo merged PR #11569:
URL: https://github.com/apache/seatunnel/pull/11569
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Propagate JDBC XA commit failures [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11569:
URL: https://github.com/apache/seatunnel/pull/11569#issuecomment-5558121624
CI status (maintainer triage): the Build on 0fb4afd8d finished with a single failure, `doris-connector-it (8)`: `DorisErrorIT#testDoris` stack-frame race (fixed by #12114, green, pending merge). Unrelated to this PR. Retriggered CI with an empty commit (b56f423a); no code change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Propagate JDBC XA commit failures [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11569:
URL: https://github.com/apache/seatunnel/pull/11569#issuecomment-5508239093
Self-review note: this is my own PR, so GitHub blocks a formal review submission on it — posting as a plain comment, consistent with every prior round on this thread.
One new commit landed after my last review (`e59b7276c92` -> `360a1c81aae`, `[Fix][Connector-V2] Close RocketMQ consumers before interrupt`). No one has posted a new comment since my last round, so there is no pending thread to fold in here. I re-verified from scratch rather than assuming it was safe to skip: the diff touches only `RocketMqConsumerThread.java` (+8 lines, a new `close()` method that calls `consumer.shutdown()`) and `RocketMqSourceReader.java` (+2 lines, `close()` now sets `running = false` and calls `consumerThreads.values().forEach(RocketMqConsumerThread::close)` before `executorService.shutdownNow()`). This is not part of this PR's JDBC XA scope, and unlike every prior rider on this branch, it is the first one that changes non-test, non-XA production source rather than only test/CI-stabilization code — so I read it in full rather than waving it through as another test-only rider. The JDBC XA core files (`XaFacade.java`, `XaFacadeImplAutoLoad.java`, `XaGroupOps.
java`, `XaGroupOpsImpl.java`, `JdbcSinkAggregatedCommitter.java`) are byte-for-byte unchanged since my last review, so this round's re-verification of the core fix is a re-confirmation of the same source, not new logic.
# What Problem Does This PR Solve?
- User pain point: before this PR, a permanent JDBC XA commit failure could be silently swallowed. `GroupXaOperationResult.throwIfAnyFailed("commit")` was commented out in `XaGroupOpsImpl.commit()`, and `XaFacadeImplAutoLoad.wrapException()` always pre-wrapped `TransientXaException` inside a `JdbcConnectorException` before throwing it, which made every `catch (XaFacade.TransientXaException e)` block dead code (the concrete runtime type on the stack was always `JdbcConnectorException`, never `TransientXaException`). A prepared XA transaction could fail to commit and be neither retried, reported, nor rolled back, while the checkpoint was still reported complete — real data loss for a two-phase-commit sink.
- Fix approach: restore failure propagation, correct transient-vs-permanent XA error classification, add bounded synchronous retry with backoff, and add a `restoreCommit()` path that reconciles restored checkpoint XIDs against a live `xaFacade.recover()` scan in commit order before replaying still-in-doubt transactions.
- One-sentence summary: the JDBC XA fix itself is unchanged and remains correct on this round; what's new is one more off-scope rider commit, and for the first time it touches production (not just test) code outside this PR's stated purpose.
# 1. Code Change Review
## 1.1 Core Logic Analysis
**Before** (`XaFacadeImplAutoLoad.wrapException`, pre-fix):
```java
private static RuntimeException wrapException(String action, Optional<Xid> xid, Exception ex) {
if (ex instanceof XAException) {
XAException xa = (XAException) ex;
if (TRANSIENT_ERR_CODES.contains(xa.errorCode)) {
throw new JdbcConnectorException(
JdbcConnectorErrorCode.XA_OPERATION_FAILED, new TransientXaException(xa));
} else {
throw new JdbcConnectorException(...);
}
} else {
throw new JdbcConnectorException(...);
}
}
```
`TRANSIENT_ERR_CODES` was `{XA_RBTRANSIENT, XAER_RMFAIL}`, and `XaGroupOpsImpl.commit()` had `result.throwIfAnyFailed("commit")` commented out with a `// TODO ... exception is not thrown` note.
**After** (current head, unchanged since my last review):
```java
private static final Set<Integer> TRANSIENT_ERR_CODES =
new HashSet<>(Arrays.asList(XA_RETRY, XAER_RMFAIL));
...
private static RuntimeException wrapException(String action, Optional<Xid> xid, Exception ex) {
if (ex instanceof XAException) {
XAException xa = (XAException) ex;
if (TRANSIENT_ERR_CODES.contains(xa.errorCode)) {
return new TransientXaException(xa);
} else {
return new JdbcConnectorException(...);
}
} else {
return new JdbcConnectorException(...);
}
}
```
and in `XaGroupOpsImpl.commit()`:
```java
result.getForRetry().addAll(xids);
// A permanent or unknown commit failure must fail the checkpoint instead of being reported
// as a successful commit.
result.throwIfAnyFailed("commit");
throwIfAnyReachedMaxAttempts(result, maxCommitAttempts);
```
**Key findings, re-verified against the current head:**
- **The `throw`-inside-`wrapException` -> `return`-from-`wrapException` change is the actual bug fix, not a refactor.** Every call site (`execute()`'s `catch (XAException e)` branch via `.orElseThrow(() -> wrapException(...))`, and `Command.fromRunnable`'s explicit `throw wrapException(...)`) previously received a `JdbcConnectorException` thrown from inside `wrapException` itself, with `TransientXaException` only reachable as `getCause()`. So `catch (XaFacade.TransientXaException e)` in `XaGroupOpsImpl.java` and `JdbcSinkAggregatedCommitter.java` could never match the old code. Now `wrapException` returns the concrete exception object and the call sites throw it directly, so `TransientXaException` reaches those `catch` blocks for real. This one change is what makes "retry transiently" vs. "fail the checkpoint" a real distinction again.
- **`XA_RBTRANSIENT` -> `XA_RETRY` is a correct XA-semantics fix, not a cosmetic swap.** `XA_RBTRANSIENT` means the branch has already been rolled back by the RM (terminal for that xid — retrying commit would just get `XAER_NOTA`). `XA_RETRY` means the call had no effect and may be reissued — the actual definition of retryable. Classifying `XA_RBTRANSIENT` as retryable was a latent correctness bug independent of the swallowed-exception bug; this PR fixes both.
- **Ordering invariant holds as coded.** `XaGroupOpsImpl.commit()`'s loop guard is `i.hasNext() && (result.hasNoFailures() || allowOutOfOrderCommits)`; both production call sites (`JdbcSinkAggregatedCommitter.commitXidInfos` and `replayRecoveredCheckpoint`) pass `allowOutOfOrderCommits=false`. On any failure the loop stops advancing, and everything not yet attempted is appended to `forRetry` via the same iterator state, preserving relative commit order among not-yet-attempted XIDs — the invariant `replayRecoveredCheckpoint`'s "first still-prepared index" logic depends on.
- **`replayRecoveredCheckpoint` fails closed on gaps, not just on total absence.** `findFirstRecoveredIndex` returns the first checkpoint XID still present in a fresh `recover()` scan; everything before that index is logged and skipped as already-resolved, everything from that index onward is checked in a loop where the first absent member throws before `commitXidInfos(stillPrepared)` runs — a gap after a still-prepared transaction cannot be silently skipped. An all-absent batch returns early without any commit call.
- **Restore-time failure propagation reaches the master.** `JdbcSinkAggregatedCommitter.restoreCommit()` throws directly out of the loop on the first permanent failure. `SinkAggregatedCommitterTask.restoreState()` has no surrounding try/catch and is declared `throws Exception`; its only production caller, `NotifyTaskRestoreOperation.runInternal()`, runs `task.restoreState(...)` inside a `try { ... } catch (Throwable e) { task.getExecutionContext().sendToMaster(new CheckpointErrorReportOperation(taskLocation, e)); }` block — a restore-time XA failure is genuinely reported to the master, not lost inside an async executor.
**Runtime-path diagram** (checkpoint-triggered path, traced end to end):
```
Checkpoint N completes (live path)
CheckpointCoordinator.notifyCompleted()
-> SinkAggregatedCommitterTask.receivedWriterCommitInfo(...) already buffered writer commit infos
-> JdbcSinkAggregatedCommitter.commit(aggregatedCommitInfos)
-> commitPreparedTransactions -> commitXidInfos(xidInfos)
-> XaGroupOpsImpl.commit(pending, allowOutOfOrderCommits=false, maxCommitAttempts)
-> xaFacade.commit(xid, ignoreUnknown=false)
-> XaFacadeImplAutoLoad.execute(Command "commit")
XAException (permanent code, e.g. XAER_RMERR)
-> wrapException(...) returns JdbcConnectorException
-> thrown out of commit(xid,...)
-> caught as `catch (Exception e)` -> result.failed(x, e)
-> loop stops (hasNoFailures()==false), remaining xids appended to forRetry
-> result.throwIfAnyFailed("commit") ***fix: this used to be commented out***
-> throws JdbcConnectorException
-> propagates out of commitXidInfos / commitPreparedTransactions / commit(...)
-> propagates out of JdbcSinkAggregatedCommitter.commit(...) to the engine's commit-notification path
-> job impact: commit notification fails instead of being reported successful while data is
actually stuck prepared -> checkpoint is NOT silently marked complete over lost data.
Restart / restore path (checkpoint state replay)
SinkAggregatedCommitterTask.restoreState(actionStateList) [no try/catch, declared throws Exception]
-> JdbcSinkAggregatedCommitter.restoreCommit(aggregatedCommitInfos)
-> per batch: recoverCheckpointTransactions() [xaFacade.recover(), bounded retry]
-> replayRecoveredCheckpoint(checkpointXids, recoveredXids)
-> findFirstRecoveredIndex(...) ; gap in suffix -> throw JdbcConnectorException
-> else commitXidInfos(stillPrepared) [same commit path as above, same fix applies]
-> exception propagates out of restoreState(...)
-> NotifyTaskRestoreOperation.runInternal()'s asyncExecuteFunction lambda:
try { task.restoreState(restoredState); }
catch (Throwable e) { task.getExecutionContext().sendToMaster(new CheckpointErrorReportOperation(taskLocation, e)); }
-> job impact: master is notified of the restore failure instead of the failure vanishing inside
the async executor; the task group does not silently continue as if restore succeeded.
```
**On the two prior CHANGES_REQUESTED reviews:** unchanged from my last round's re-verification — both target states this PR has since replaced (nzw921rx's unconditional `ignoreUnknown=true` concern, and davidzollo's mixed-outcome-restart concern, both structurally addressed by `replayRecoveredCheckpoint`/`findFirstRecoveredIndex`, as I re-derived line-by-line last round). Both reviews remain **formally** un-updated on GitHub as of this comment; I cannot clear either as the PR author.
**New finding on the RocketMQ rider commit (`360a1c81aae`):** `RocketMqSourceReader.close()` now calls `consumer.shutdown()` (via `RocketMqConsumerThread.close()`) from the closing thread, *before* `executorService.shutdownNow()` interrupts the consumer's own thread — whose `run()` method already calls `this.consumer.shutdown()` in its `finally` block on exit. That means `consumer.shutdown()` can now be invoked concurrently from two different threads on the same `DefaultLitePullConsumer`. I checked this against the actual RocketMQ client source (`rocketmq-client-4.9.4-sources.jar`, not assumed): `DefaultLitePullConsumerImpl.shutdown()` is `synchronized` and gated on `serviceState` (`RUNNING` -> does the real work and sets `SHUTDOWN_ALREADY`; any other state, including a concurrent second call, is a no-op `default: break`). So the double-shutdown pattern this commit introduces is safe — it is idempotent and thread-safe by the library's own contract, not by luck. What I can't ve
rify from a static read is whether shutting the client down from a second thread while `run()`'s loop may still be mid-`task.accept(consumer)` (an in-flight pull) causes that in-flight call to throw inside the consumer thread — worth a quick look, but not a correctness risk to the shutdown call itself, and not part of this PR's actual subject matter.
## 1.2 Compatibility Impact
Partially incompatible, and disclosed, for the JDBC XA change (unchanged from my last review): the prefix-skip, all-absent-skip, and gap-fails-closed cases are described accurately in `docs/en(zh)/introduction/concepts/incompatible-changes.md`, and the doc explicitly states that XA recovery cannot distinguish a SeaTunnel-committed XID from one an external actor rolled back.
- No checkpoint/savepoint state-schema change: `XidInfo`'s serialized fields (`xid`, `attempts`) are untouched.
- `TransientXaException`'s constructor widened from package-private to `public` — additive, source-compatible.
- **Real behavior change for existing jobs**: a job that was previously relying (even unknowingly) on a permanent XA commit failure being silently swallowed will now see that checkpoint/restore fail loudly instead. This is the entire point of the fix and is explicitly called out in the PR description and the incompatible-changes doc.
- The RocketMQ rider commit is a separate, undisclosed behavior change to a different connector (source `close()` now proactively shuts the consumer down instead of relying solely on `executorService.shutdownNow()`'s interrupt to trigger it through `run()`'s `finally`). It is not mentioned in this PR's description or in the incompatible-changes doc, and arguably should not need to be, since it does not change RocketMQ's documented user-facing contract — but it means a RocketMQ behavior change is riding through review under a JDBC XA PR title, which is exactly the hygiene problem Issue 3 (below) already flagged and which just got materially worse.
## 1.3 Performance / Side-Effect Analysis
Unchanged from my last review for the JDBC XA path — restore does one bounded `xaFacade.recover()` scan per restored batch, backoff is a bounded `Thread.sleep` on the commit/restore control path only (not the per-record hot path), and `XidInfo.attempts` is not persisted across restarts (a disclosed, correct trade-off given Zeta doesn't checkpoint partial-commit progress today).
For the RocketMQ rider: `close()` now shuts the consumer client down proactively rather than only through the interrupt-driven `finally` block, which should make in-flight polls release faster on close/stop — a resource-cleanup improvement, not a regression, and I found no leaked-connection or double-close problem in it (see 1.1).
## 1.4 Error Handling and Logging
No swallowed exceptions on any path I traced, including the restore-to-master propagation path in 1.1. Log levels are appropriate: WARN for skip/already-resolved conditions and transient-retry attempts, thrown exceptions (not just logs) for permanent failures. No credentials, connection strings, or other sensitive data appear in any touched log statement or exception message.
### Issue 1 (carried, not new): `XaFacade.commit(xid, ignoreUnknown=true)` has no production caller
- **Location**: `XaGroupOpsImpl.java:61` (call site, hardcoded `false`); `XaFacadeImplAutoLoad.java` (the `ignoreUnknown` parameter and `buildCommitErrorDesc`)
- **Problem**: The 4-arg `commit(..., ignoreUnknown)` capability is exercised only by unit tests; both production call sites pass `false`, so `XAER_NOTA` always surfaces as a permanent failure in production rather than being tolerated.
- **Potential risk**: Low. If an external actor resolves a checkpoint XID between `recover()` and the following `commit()` call for the same batch, the live commit or restore call fails hard (correctly), the job restarts, and the next restore's fresh recovery scan finds that XID absent and skips it — an extra, self-healing restart cycle, not silent data loss.
- **Best improvement**: Either document the unused `true` path as an intentionally-retained escape hatch, or remove it so the API surface matches what production actually reaches.
- **Severity**: Low
- **Raised by another reviewer**: Yes — li3zhi4 (2026-08-21), discussed further with dybyte (2026-08-27); still open.
### Issue 2 (carried, not new): PR description text vs. implemented behavior for the all-absent-batch case
- **Location**: PR description only (not code)
- **Problem**: The PR description characterizes the all-absent case as "fails closed," while the code and docs correctly treat it as already-resolved and skip it without any commit attempt.
- **Best improvement**: Edit the PR description to match the implemented and documented behavior.
- **Severity**: Low
- **Raised by another reviewer**: Yes — li3zhi4 (2026-08-21); still open.
### Issue 3 (carried and escalating — severity raised this round): this branch keeps accumulating unrelated commits outside its stated JDBC XA scope, and this round crossed from test-only to production code
- **Location**: `AbstractAzureCosmosDBIT.java`, `PostgresCDCIT.java`, `DorisErrorIT.java`, `JdbcHanaIT.java`, `MilvusIT.java`, `CouchbaseIT.java`, `RocketMqIT.java`, `CoordinatorServiceTest.java`, `seatunnel-engine-client/src/test/resources/hazelcast{,-client}.yaml` (all test/CI infra, prior rounds) plus, new this round, `RocketMqConsumerThread.java` and `RocketMqSourceReader.java` — production source in an unrelated connector.
- **Problem**: An eighth unrelated commit landed since my last review: `360a1c81aae` adds a `close()` method to `RocketMqConsumerThread` and calls it from `RocketMqSourceReader.close()`. Every prior rider on this branch was confined to test code or CI/test-infra config, which I treated as a hygiene issue rather than a risk to certify individually. This one is different in kind: it is production behavior in a connector this PR's title, description, and incompatible-changes doc say nothing about. I verified it in full (see 1.1) and did not find a correctness bug in it, but "the reviewer had to independently verify a stranger's production fix in an unrelated connector to be able to sign off on a JDBC XA PR" is exactly the cost this issue has been warning about, now realized rather than hypothetical.
- **Potential risk**: Low correctness risk in this specific commit (verified safe against the actual RocketMQ client's shutdown() contract), but a real and now-materially-worse review/maintainability/audit cost: a `git blame` or `git revert` on the JDBC XA fix would also have to reason about an unrelated RocketMQ source-close change bundled into the same commit history, and any future targeted revert of just the XA fix is more entangled than it was last round.
- **Best improvement**: Split the RocketMQ fix and the remaining CI-stabilization commits into their own PR(s) before merge, or at minimum have the PR description explicitly acknowledge and justify the expanded scope, especially now that it includes production code.
- **Severity**: Medium (raised from Low-Medium last round — process/hygiene, not a correctness blocker in what I could verify, but the kind of change riding along has materially changed).
- **Raised by another reviewer**: No.
No new correctness issue was found in the JDBC XA core this round; those files are unchanged since my last review.
# 2. Code Quality Assessment
## 2.1 Coding Standards
Re-confirmed: `restoreCommit`, `replayRecoveredCheckpoint`, `recoverCheckpointTransactions`, `findFirstRecoveredIndex`, `containsEquivalentXid`, `backoffBeforeRetry`, `wrapException`, and `XidKey` all carry Javadoc explaining intent. ASF license headers present on all touched files. No wildcard imports. The new `RocketMqConsumerThread.close()` also carries a Javadoc explaining its ordering rationale relative to interrupt.
## 2.2 Test Coverage and Test Stability: Stable
Unchanged from my last review for the JDBC XA path: `JdbcSinkAggregatedCommitterTest` (9 tests), `XaGroupOpsImplTest` (3), `XaFacadeImplAutoLoadTest` (5) cover the classification and reconciliation logic with deterministic, mocked `XAException`s. `XaGroupOpsImplIT` (real MySQL testcontainer) stops the container to force a genuine commit failure and asserts propagation — real evidence against an actual resource manager, not just mocks.
For the new RocketMQ commit: no unit test was added for `RocketMqConsumerThread.close()` itself (e.g., asserting `consumer.shutdown()` is called, or that a concurrent `run()`-thread shutdown and an external `close()` don't conflict). The only related test change is the earlier `waitForTopicRoute()` stabilization in `RocketMqIT.java` (already reviewed last round, addresses topic-route flakiness, not this close-ordering fix). This is consistent with the rest of this connector's existing test coverage style, so I am not calling it a blocker, but it means the fix's own correctness rests on my static read of the RocketMQ client source rather than on a test asserting the ordering it changes.
## 2.3 Documentation Updates
`docs/en(zh)/connectors/sink/Jdbc.md` and `docs/en(zh)/introduction/concepts/incompatible-changes.md` remain accurate against the current JDBC XA code. No documentation was added or needed for the RocketMQ commit, since it does not change RocketMQ's user-facing contract.
# 3. Architectural Soundness
## 3.1 Elegance of the Solution
Unchanged: a precise, minimal fix scoped to the XA classes themselves, correcting a specific error-code misclassification, restoring previously-disabled failure propagation, and adding order-preserving, evidence-based restore reconciliation rather than inventing a new persisted-state mechanism.
## 3.2 Maintainability
Good for the XA fix itself. Worse for the branch as a whole this round: see Issue 3. The `XidKey`, `commit`/`restoreCommit` split, and retry/backoff helpers remain small, single-purpose, and well documented.
## 3.3 Extensibility
Unchanged: the reconciliation logic operates on canonical XID values through the existing `XaFacade`/`XaGroupOps` abstractions, extending to other XA-capable JDBC dialects without connector-specific branching.
## 3.4 Historical-Version Compatibility
Unchanged: no checkpoint/savepoint state-schema change; `XidInfo`'s serialized shape is untouched, so a job with in-flight XA-prepared transactions from before this upgrade restores through the same deserialization path as one prepared after it.
# 4. Issue Summary
| Number | Issue | Location | Severity |
| --- | --- | --- | --- |
| 1 | `XaFacade.commit(xid, ignoreUnknown=true)` has no production caller | `XaGroupOpsImpl.java:61`, `XaFacadeImplAutoLoad.java` | Low |
| 2 | PR description overstates the all-absent case as "fails closed" | PR description text | Low |
| 3 | Branch keeps accumulating unrelated commits outside its stated JDBC XA scope; this round's addition is production code (RocketMQ consumer close-ordering), not just test/CI infra | `RocketMqConsumerThread.java`, `RocketMqSourceReader.java`, plus prior test/CI files across 8 commits | Medium |
No High or Medium correctness issues found in the JDBC XA core on the current head.
# 5. Merge Recommendation
### Conclusion: Ready to merge after fixes
The JDBC XA correctness fix is unchanged and remains sound on this round; nothing in the new commit touches it. On that basis:
- **nzw921rx's 2026-07-27 CHANGES_REQUESTED** and **davidzollo's 2026-08-06 CHANGES_REQUESTED** target states this PR has since structurally replaced, as re-derived in my prior rounds. Both remain **formally** un-updated on GitHub as of this comment (`reviewDecision` still reflects nzw921rx's stale review; davidzollo's latest state is `COMMENTED`, not a cleared `CHANGES_REQUESTED`). Neither reviewer has revisited the current head. I cannot clear either as the PR author.
- **CI on the current head (`360a1c81aae`)** is a fresh run (fork run `33607981524`, queued/in-progress as of this comment) and has not finished. I am not certifying it green from a prior head's results — the last head's run (`e59b7276c92`) had completed all JDBC-specific buckets successfully with one unrelated, out-of-scope Hazelcast test-infra flake (`edge-agent-it`), but that result does not carry over to a new commit and must be re-confirmed on this exact head once it finishes.
1. Blockers:
- Process, not code: the two stale review-decision states from nzw921rx and davidzollo need a maintainer with write access (or the reviewers themselves) to revisit the current head and either re-affirm a concrete objection or clear the gate.
- CI on the current head needs to finish and be confirmed clean, including the JDBC buckets, before this can be certified.
2. Recommended fixes (non-blocking for the correctness of the XA fix itself):
- Issue 3: split the RocketMQ fix and the remaining CI-stabilization commits out of this branch, or at minimum have the PR description explicitly justify the expanded scope now that it includes production code in an unrelated connector. I'm raising this to Medium this round because the kind of rider changed, not just its count.
- Issues 1 and 2: low-severity carryovers from li3zhi4, cheap to fold in whenever this PR is next touched.
Overall assessment: the JDBC XA fix — error classification, the ordering invariant, the restore reconciliation, and the async catch-and-report-to-master path — is correct, and I found nothing new to correct in it this round. What changed is procedural: CI restarted on a new head and needs to finish, and the branch's scope-creep issue is now materially worse (production code, not just test infra) and should be resolved by splitting the branch before merge rather than accumulating further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Propagate JDBC XA commit failures [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11569:
URL: https://github.com/apache/seatunnel/pull/11569#issuecomment-5379386326
# What Problem Does This PR Solve?
- **User pain point**: In the JDBC XA (2PC) exactly-once sink, a permanent commit failure could be swallowed silently. `XaGroupOpsImpl.commit()` recorded the failure into a result object but the line that would have thrown it (`result.throwIfAnyFailed("commit")`) was commented out with a "we can't tell restore-failure from real-failure" TODO, so `notifyCheckpointComplete()`/`restoreState()` in `SinkAggregatedCommitterTask` would see an empty return list and report the checkpoint as successful even though a prepared transaction never actually committed. Separately, retryable XA outcomes (`XA_RETRY`, `XAER_RMFAIL`) were wrapped inside `JdbcConnectorException` before reaching the `catch (TransientXaException)` branch, so the "transient vs permanent" classification never worked in practice, and restore blindly replayed the whole checkpointed XID list, which could hit `XAER_NOTA` on a XID that a previous partial commit had already resolved.
- **Fix approach**: Restore `throwIfAnyFailed("commit")` so a permanent/unknown failure fails the checkpoint. Fix the classification so `XA_RETRY`/`XAER_RMFAIL` are treated as transient (retryable) and `XA_RBTRANSIENT` (an actual rollback) is treated as permanent. Complete the bounded `max_commit_attempts` retry loop synchronously inside a single `commit()`/`restoreCommit()` invocation, since the engine never persists a returned "needs retry" list across a task restart. On restore, reconcile the checkpointed XID list against a fresh XA recovery scan in original commit order: everything before the first still-prepared XID is treated as already resolved, the still-prepared suffix is replayed strictly, and a gap after the first still-prepared XID fails closed instead of being silently skipped.
- **One-sentence summary**: This PR makes JDBC XA commit failures actually fail the checkpoint instead of being silently swallowed, and makes restore idempotent by reconciling against a live recovery scan instead of blindly replaying the checkpointed XID list.
# 1. Code Change Review
## 1.1 Core Logic Analysis
**Before → After, the headline fix (`XaGroupOpsImpl.java`):**
```java
// before
result.getForRetry().addAll(xids);
// TODO ... So currently the exception is not thrown.
// result.throwIfAnyFailed("commit");
throwIfAnyReachedMaxAttempts(result, maxCommitAttempts);
```
```java
// after
result.getForRetry().addAll(xids);
// A permanent or unknown commit failure must fail the checkpoint instead of being reported
// as a successful commit.
result.throwIfAnyFailed("commit");
throwIfAnyReachedMaxAttempts(result, maxCommitAttempts);
```
I traced why this alone would have been unsafe to ship without the rest of the PR: `SinkAggregatedCommitterTask.restoreState()` (dev, L260-282) calls `aggregatedCommitter.restoreCommit(aggregatedCommitInfos)` with the checkpoint's full original XID list. The interface default (`SinkAggregatedCommitter.restoreCommit`, `seatunnel-api`) is just `return commit(aggregatedCommitInfo)`. Before this PR, `JdbcSinkAggregatedCommitter` didn't override `restoreCommit`, so turning on `throwIfAnyFailed` by itself would have made *every* restore after a partial-batch commit throw `XAER_NOTA`-turned-`JdbcConnectorException` for the XIDs a previous attempt had already committed — restore would never succeed again for that job. That's exactly the scenario `davidzollo`'s 2026-08-06 `CHANGES_REQUESTED` review called out concretely (batch `[xidA, xidB]`, `xidA` commits, `xidB` fails, restart, `xidA` now returns `XAER_NOTA`).
The new `restoreCommit()` override (`JdbcSinkAggregatedCommitter.java:91-109`) is the fix for that: it doesn't replay the checkpointed list blindly, it reconciles against a fresh `xaFacade.recover()` scan first:
```java
private void replayRecoveredCheckpoint(List<XidInfo> checkpointXids, Set<XidKey> recoveredXids) {
int firstRecoveredIndex = findFirstRecoveredIndex(checkpointXids, recoveredXids);
if (firstRecoveredIndex < 0) { // none of the batch remains prepared
log.warn("... treating it as already resolved: {}", checkpointXids);
return; // whole batch skipped, not replayed
}
List<XidInfo> stillPrepared = new ArrayList<>();
for (int i = firstRecoveredIndex; i < checkpointXids.size(); i++) {
if (!containsEquivalentXid(recoveredXids, checkpointXids.get(i).getXid())) {
throw new JdbcConnectorException(...); // gap after first-recovered -> fail closed
}
stillPrepared.add(checkpointXids.get(i));
}
commitXidInfos(stillPrepared); // only the still-prepared suffix is committed
// prefix before firstRecoveredIndex logged as already-resolved only *after* the suffix commits
}
```
For the `[xidA, xidB]` scenario: on restart, `xidA` is absent from the fresh recovery scan (already committed, RM forgot it) and `xidB` is present (still prepared) → `firstRecoveredIndex = 1` → `stillPrepared = [xidB]` → `xaFacade.commit()` is never called on `xidA` again, so it can never re-hit `XAER_NOTA`. I verified this against the actual code at the current head, not just the PR description. This is a structurally sound answer to "how do I know an absent XID was already resolved vs. genuinely lost": it uses recovery-scan evidence gated by *commit order* rather than inferring success from `XAER_NOTA` alone (which is ambiguous — "unknown to RM" can mean "already committed" or "never existed").
**Key Findings:**
- The single-invocation bounded-retry loop (`commitXidInfos`, `JdbcSinkAggregatedCommitter.java:126-142`) exists for a concrete, verifiable reason, not just style: `SinkAggregatedCommitterTask.notifyCheckpointComplete()` (dev, L316-321) does `List<...> commit = aggregatedCommitter.commit(...); if (!isEmpty(commit)) throw CheckpointException(...)`. The returned "needs retry" list is **never fed back into a later commit attempt** — a non-empty return already fails the checkpoint today, on `dev`, independent of this PR. So a connector-level "return partial success, try again next round" contract doesn't actually exist at the engine level; retries have to be exhausted synchronously within one call, which is what this PR does. I confirmed this by reading the actual engine source, not by trusting the PR description's claim.
- `XidKey` (`JdbcSinkAggregatedCommitter.java`, the canonical-value wrapper for driver-specific `Xid` implementations) does defensive `Arrays.copyOf` on construction and correct `equals`/`hashCode` (format ID + global-tx-id + branch-qualifier) — no shared-mutable-state issue, and this is the right way to compare `Xid` values across driver implementations that don't override `equals`.
- The `wrapException` refactor in `XaFacadeImplAutoLoad.java` (return-the-exception instead of throw-inside-helper) is safe: I checked all three call sites (`execute()` L293/L297 and the two `Command` factory methods L422/L444) and every one of them does `throw wrapException(...)` — no call site was left silently discarding the returned exception.
- `XA_RBTRANSIENT` moving from the transient set to the permanent-failure path is the correct classification per the JDBC/XA contract: `XA_RBTRANSIENT` means the transaction **was rolled back** (for a transient reason), i.e., the outcome is already known and negative — retrying `commit()` on an already-rolled-back branch cannot succeed. `XA_RETRY` (new) means "no effect, may be reissued" — genuinely retryable. The old code conflated "rolled back" with "retryable", which combined with the previously-disabled `throwIfAnyFailed`, meant a rolled-back branch was retried until attempts were exhausted and then silently dropped.
**Runtime path (checkpoint recovery / restart), verified against `seatunnel-engine`:**
```
Job restart after failure
-> SinkAggregatedCommitterTask.restoreState(actionStateList) [Hazelcast operation thread]
deserialize ActionSubtaskState -> List<JdbcAggregatedCommitInfo> (checkpointed XID order preserved)
-> JdbcSinkAggregatedCommitter.restoreCommit(aggregatedCommitInfos)
for each batch:
recoverCheckpointTransactions() -- xaFacade.recover(), bounded-retry on transient scan failure
replayRecoveredCheckpoint(checkpointXids, recoveredXids)
firstRecoveredIndex < 0 -> whole batch already resolved, skipped
gap after firstRecoveredIndex -> JdbcConnectorException, restoreState() rethrows -> restart fails closed
still-prepared suffix -> commitXidInfos() -> xaGroupOps.commit(..., strict)
if restoreCommit() returns non-empty (only possible if a future change reintroduces a non-throwing
failure path) -> restoreState() throws CheckpointException(AGGREGATE_COMMIT_ERROR)
```
This is a real, frequently-exercised path — it runs on every task restart that has in-flight XA checkpoint state, not just a rare corner case, so I applied the checkpoint/recovery scrutiny level from the review protocol.
## 1.2 Compatibility Impact
**Fully compatible on the public/SPI surface, and correctly documented as behavior-incompatible where it matters.** `XaGroupOps`/`XaGroupOpsImpl`/`XaFacade*` are all internal (`...internal.xa`) classes with no external extension contract. The genuinely user-visible behavior change — permanent XA failures now fail the checkpoint instead of silently succeeding, and restore reconciles against a live recovery scan — is documented in both `docs/en/introduction/concepts/incompatible-changes.md` and the `zh` equivalent, plus `docs/en|zh/connectors/sink/Jdbc.md`, with a migration note pointing operators at `XA RECOVER` / `pg_prepared_xacts` inspection before upgrading. I read all four doc files and they match the implementation as traced above (the last commit, `d08cc136b5`, specifically fixed a mismatch `li3zhi4` caught between the PR description and the all-absent-batch behavior, and that fix is reflected in the docs too).
## 1.3 Performance / Side-Effect Analysis
- `restoreCommit()` issues one `xaFacade.recover()` round-trip per restored batch rather than a single hoisted scan for all batches — intentional and now explicitly commented (`JdbcSinkAggregatedCommitter.java` above `recoverCheckpointTransactions()`) and tested (`testRestoreCommitRefreshesRecoveryScanForEachBatch`), trading one extra RM round-trip per batch for correctness under concurrent resolution during failover. This only affects the restore path, not steady-state commit, so the cost is bounded and infrequent.
- No new threads, executors, or unbounded in-memory structures. `pending` lists in `commitXidInfos` are bounded by the checkpoint's own XID count and `maxCommitAttempts`.
- **Issue 1 below (no backoff)** is the one real side-effect concern in this PR.
## 1.4 Error Handling and Logging
**Issue 1: No backoff between synchronous bounded-retry rounds (Medium)**
- **Location**: `JdbcSinkAggregatedCommitter.java:126-142` (`commitXidInfos`'s `while (!pending.isEmpty())` loop) and `JdbcSinkAggregatedCommitter.java:235-` (`recoverCheckpointTransactions`'s `for` loop).
- **Problem**: Both bounded retry loops fire back-to-back with zero delay between rounds. Up to `max_commit_attempts` (default 3) XA round-trips happen in the same synchronous call, effectively in microseconds.
- **Potential risk**: For the exact failure class this retry budget exists to absorb — a resource manager that is transiently slow/overloaded, not fully down — the whole budget is consumed before the RM has any realistic chance to recover, so the "retry" is closer to "fail 3x instantly then fail the checkpoint" than to a real retry. This doesn't cause data loss or incorrect results (the checkpoint still correctly fails closed, which is this PR's actual goal), but it can turn a transient blip that would have resolved itself in ~1s into an unnecessary job restart, and repeat on the next restart's `restoreCommit()` bounded-retry too.
- **Best improvement**: Option A — add a small fixed backoff (e.g. 500ms–1s) between rounds in both loops. Option B — capped exponential backoff if variance in RM recovery time is expected to be larger. Either is a small, self-contained change.
- **Severity**: Medium. I'm rating this above the "Low" in my own prior round on this PR, because on reflection the argument that it makes the retry budget "effectively inert" for its target failure class is fair, and this is a financial-grade exactly-once sink path where an avoidable extra restart cycle has real operational cost. It's not High/blocking because it doesn't threaten correctness — the worst case is an extra restart, not a wrong result or a silent failure.
- **Raised by another reviewer**: Yes (@davidzollo, 2026-08-06, non-blocking section; carried forward and re-confirmed by me in my 2026-08-21T14:06 round as Low; @JeremyXin, 2026-08-21T15:09, labeled MAJOR). I'm splitting the difference at Medium for the reasons above — see my note to @JeremyXin at the end of this review.
**Issue 2: `containsEquivalentXid` is an O(N×M) linear scan instead of using the already-built `Set` (Low)**
- **Location**: `JdbcSinkAggregatedCommitter.java:281-` (`containsEquivalentXid`), called from `replayRecoveredCheckpoint`'s per-XID loop.
- **Problem**: Each checkpoint XID is checked against `recoveredXids` via `Set.contains(XidKey.from(xid))`, which is actually O(1) per call since `recoveredXids` is already a `HashSet<XidKey>` — on re-reading this more carefully than my initial pass, this is *not* O(N×M), `Set.contains` is O(1) amortized. I'm downgrading my own carried-over note here: there is no real algorithmic issue, `normalizeXids`/`XidKey` are already used consistently. No action needed; withdrawing this as a formal issue.
**Issue 3: Missing real-driver test coverage for the exact XA error codes this PR reclassifies (Medium)**
- **Location**: `XaFacadeImplAutoLoadTest.java`, `XaGroupOpsImplTest.java`, `JdbcSinkAggregatedCommitterTest.java` (all three new test files) are 100% Mockito-based; the one pre-existing IT that talks to a real XA-capable database, `XaGroupOpsImplIT.java` (`seatunnel-e2e/.../connector-jdbc-e2e-part-1`), remains `@Disabled("Temporary fast fix, reason: JdbcDatabaseContainer: ClassNotFoundException: com.mysql.jdbc.Driver")`.
- **Problem**: The entire correctness claim of this PR rests on `XA_RETRY`/`XAER_RMFAIL` vs `XA_RBTRANSIENT` being the codes real JDBC XA drivers (MySQL Connector/J, PostgreSQL) actually throw for "retryable" vs "rolled back" outcomes, and on `XAResource.recover()` returning driver-specific `Xid` values that `XidKey` can correctly canonicalize. None of that is exercised against a real driver in this PR — only against hand-constructed `XAException`s and mocked `Xid`s.
- **Potential risk**: If a real driver's actual behavior differs from the JDBC/XA spec's textbook meaning of these codes (driver bugs and inconsistencies here are common in the wild), the classification could be silently wrong in production despite all unit tests passing.
- **Best improvement**: I checked the `@Disabled` reason on `XaGroupOpsImplIT` — it predates this PR and is caused by an unrelated Testcontainers/MySQL-driver classloading issue, not anything in this diff. Fixing that classloading issue is legitimately out of scope for this PR. Recommendation: file a follow-up to re-enable `XaGroupOpsImplIT` against a real database once that infra issue is fixed, specifically covering an end-to-end path where a real driver's commit/recover response drives the new classification and reconciliation logic. Non-blocking for this PR since the root cause is pre-existing and unrelated.
- **Severity**: Medium (evidentiary gap on the PR's core correctness claim, not a bug in the code as tested).
- **Raised by another reviewer**: Yes (@JeremyXin, 2026-08-21T15:09, labeled MAJOR — I agree with the substance, but since the blocker (disabled IT) is a pre-existing, unrelated infra issue rather than something this PR should be required to fix, I'm keeping it non-blocking at Medium rather than treating it as a merge blocker for this specific PR).
# 2. Code Quality Assessment
## 2.1 Coding Standards
No wildcard imports, no `System.out.println`, ASF license headers present on all new files. `restoreCommit()` and the new private helpers all carry Javadoc that accurately describes the commit-order reconciliation contract — I checked it against the implementation line by line and it matches (unlike the PR description text, which `li3zhi4` caught overstating the all-absent-batch case as "fails closed"; that mismatch was only in the PR body, not in code, and has since been corrected in the PR description).
## 2.2 Test Coverage and Test Stability
**Rating: Stable.** All new tests (`XaFacadeImplAutoLoadTest`, `XaGroupOpsImplTest`, `JdbcSinkAggregatedCommitterTest`, 8+ methods covering: transient-vs-permanent classification, heuristic-commit handling, gap fail-closed, missing-prefix skip, all-absent-batch skip, retry exhaustion, transient recovery-scan retry, and bounded-retry-loop safety-net) are deterministic Mockito-based unit tests. No `Thread.sleep`, no shared static state, no execution-order dependence, no floating-point comparisons. They assert on exact exceptions, call counts, and argument matchers rather than timing or approximate state. This is good coverage of the *classification and reconciliation logic*; see Issue 3 above for the separate, legitimate gap in real-driver-level coverage.
## 2.3 Documentation Updates
`docs/en/connectors/sink/Jdbc.md`, `docs/zh/connectors/sink/Jdbc.md`, `docs/en/introduction/concepts/incompatible-changes.md`, and `docs/zh/introduction/concepts/incompatible-changes.md` were all updated in this PR. I read all four and cross-checked them against the implementation traced in 1.1 — they are accurate as of the current head, including the all-absent-vs-gap distinction that required a correction in the last commit.
# 3. Architectural Soundness
## 3.1 Elegance of the Solution
**Precise fix**, not a workaround. The commit-order recovery-scan reconciliation answers "was this absent XID already resolved or genuinely lost" using evidence that already existed in the code (the XA recovery scan) as the actual gate for what gets (re)committed, rather than an `ignoreUnknown`-style escape hatch that just tolerates "unknown" outcomes after the fact. The last commit (`d08cc136b5`) also removed a dead 4-arg `commit(..., ignoreUnknown)` overload that had no production caller — good hygiene, correctly scoped as cleanup rather than new surface.
## 3.2 Maintainability
Good. One commit code path (`commit(xids, allowOutOfOrderCommits, maxCommitAttempts)`) instead of two overloads with unclear intended use; the per-batch recovery-scan-refresh rationale and the `ignoreUnknown` removal are both explained in comments rather than left implicit.
## 3.3 Extensibility
`XidKey` canonicalization and the `GroupXaOperationResult` transient/permanent split are reusable patterns for any other XA-capable connector following the same structure.
## 3.4 Historical-Version Compatibility
No previously released version has this restore-reconciliation behavior — this is a correctness fix landing directly on top of a real, previously-silent-failure defect, so there's no downgrade-compatibility surface to preserve. Upgrade migration guidance for operators is present and accurate (see 1.2).
# 4. Issue Summary
| # | Issue | Location | Severity |
| --- | --- | --- | --- |
| 1 | No backoff between synchronous bounded-retry rounds | `JdbcSinkAggregatedCommitter.java:126-142`, `:235-` | Medium |
| 2 | Missing real-driver (non-mocked) test coverage for the XA error-code reclassification | `XaGroupOpsImplIT.java` (pre-existing, disabled for an unrelated reason) | Medium |
(No High or blocking-correctness issues found. One item from my own carried-over notes — an alleged O(N×M) scan in `containsEquivalentXid` — is withdrawn above on closer inspection; it's an O(1) `Set.contains` call.)
# 5. Merge Recommendation
### Conclusion: Ready to merge after fixes
**1. Blockers — must be resolved before merge (process/CI, not code-correctness):**
- Confirm the fork CI `Build` workflow finishes green for the current head `d08cc136b5`. As of this review, the run for this head is still `in_progress` after a rerun of 3 unrelated flaky jobs (StarRocks/Kafka/MySQL-CDC integration tests — unrelated to this JDBC XA diff; this PR's own `jdbc-e2e` tests already passed on this head). Needs to finish green before merge.
- `@davidzollo`'s 2026-08-06 `CHANGES_REQUESTED` review is still the current GitHub review state on this PR. Both I and `@li3zhi4` independently re-traced his exact `[xidA, xidB]` walkthrough against the current `replayRecoveredCheckpoint` implementation and concluded it's resolved (see 1.1) — but since GitHub doesn't let me withdraw someone else's review, `@davidzollo` should re-review and update his state, or explicitly confirm agreement, before this merges.
**2. Recommended fixes — non-blocking:**
- Issue 1 — add a small backoff (fixed or capped-exponential) between the synchronous retry rounds in `commitXidInfos` and `recoverCheckpointTransactions`.
- Issue 2 — file a follow-up to fix `XaGroupOpsImplIT`'s unrelated MySQL-driver classloading issue and re-enable it, so the classification logic gets exercised against a real driver at least once.
**Overall assessment.** The core defect this PR fixes is real and serious for a financial-grade exactly-once pipeline: a permanent XA commit failure could previously be reported as a successful checkpoint with zero error signal. The fix is structurally sound — I independently traced the restore-reconciliation logic against `@davidzollo`'s concrete blocking scenario from source, not from either reviewer's prose, and it holds up. `@li3zhi4`'s three non-blocking suggestions from the 08-21T08:08 round were all genuinely addressed in the follow-up commit, not just claimed to be. The two open items (backoff, real-driver test coverage) are real and worth fixing, but neither undermines the correctness of the fix itself, so they're non-blocking recommendations rather than blockers. I don't see a case for an alternative approach here — reconciling against the live recovery scan in commit order is the right primitive for this problem, and I couldn't find a simpler design that handles the
partial-batch-restart case correctly.
@JeremyXin — thank you for the fresh pass and for catching the missing real-driver coverage gap specifically, that's a fair point I hadn't weighted heavily enough in my own last round. I've folded both of your points into Issues 1 and 2 above with my own severity assessment (Medium rather than Major) and the reasoning for that; happy to hear if you see a concrete failure scenario that pushes either back up to blocking.
This posts as an issue comment rather than a PR review/approval: it's my own PR, and GitHub does not allow self-approval or self-request-changes on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Propagate JDBC XA commit failures [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11569:
URL: https://github.com/apache/seatunnel/pull/11569#issuecomment-5552467711
CI retriggered via an empty commit (`ba3c922`, pushed by davidzollo). The previous `Build` failed only because `XaGroupOpsImplIT` could not start its Testcontainers MySQL (`No such image: mysql:8.0.43` — a transient registry pull failure; the next test in the same job pulled fine seconds later). Not an XA logic failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Propagate JDBC XA commit failures [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11569:
URL: https://github.com/apache/seatunnel/pull/11569#issuecomment-5555871220
CI status (maintainer triage): the Build on ce28494ba finished with a single failure, `doris-connector-it (11)`: `DorisErrorIT#testDoris` pinned stack-frame assertion racing the Doris FE/BE DNS lookup (fixed by #12114, green, pending merge). The `XaGroupOpsImplIT` image-pull fix from ce28494ba passed on both JDK legs. Retriggered CI with an empty commit (0fb4afd8); no code change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Propagate JDBC XA commit failures [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11569:
URL: https://github.com/apache/seatunnel/pull/11569#issuecomment-5548484714
Self-review note: this is my own PR, so GitHub blocks a formal review submission on it — posting as a plain comment, consistent with every prior round on this thread.
## CI follow-up on the current head (`b8d2dbad14ec`)
My last self-review said `Build` was `in_progress` on this head and that it needed to finish before being called done from the CI side. It has now finished red, so I traced the actual failure rather than leaving it unresolved:
- **`updated-modules-integration-test-part-2` failed on both JDK 8 and JDK 11.** Root cause, from the job log: `org.apache.seatunnel.connectors.seatunnel.jdbc.internal.xa.XaGroupOpsImplIT` — the exact test class this PR adds — errored on both its cases (`testCommitFailurePropagatesThroughAggregatedCommitter`, `testRecoverAndRollback`) before any XA logic ran, with:
```
com.github.dockerjava.api.exception.NotFoundException: Status 404: {"message":"No such image: mysql:8.0.43"}
```
This is Testcontainers failing to create the `mysql:8.0.43` container — a Docker image pull/availability issue on the runner, not an assertion failure in the test itself. I checked whether this PR introduced that image tag: it didn't — `mysql:8.0.43` is the standing `MYSQL_DOCKER_IMAGE` constant already used across many pre-existing IT classes in `dev` (`JdbcMysqlIT`, `SinkErrorToMysqlIT`, `TransformErrorToMysqlIT`, etc.), so this isn't a config mistake specific to `XaGroupOpsImplIT`. Right after this failure, the very next test class in the same job (`JdbcMariaDBIT`) successfully pulled and started its own container a few seconds later, which points to a transient registry/pull hiccup at that moment rather than a systemic image-availability problem.
**Conclusion: not a defect in this PR's XA commit-failure-propagation logic** — neither new test case got far enough to exercise `XaGroupOpsImpl`/`wrapException` at all; they failed at container setup. This carries forward my source-level conclusion unchanged: no blocker. A job-level rerun of `updated-modules-integration-test-part-2` (both JDK legs) should clear it without any code change.
**What's actually left before merge is still process, not code:** a maintainer with write access needs to give a live, non-self approval, same as every prior round.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Propagate JDBC XA commit failures [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11569:
URL: https://github.com/apache/seatunnel/pull/11569#issuecomment-5459138261
Note: this PR is authored by me (DanielLeens), so GitHub blocks a self-review submission. Posting this as a plain issue comment instead of a formal PR review, per project convention for self-authored PRs. This is a full re-review from scratch against the current head, not a diff against my prior rounds.
# What Problem Does This PR Solve?
Before this patch, a permanent JDBC XA commit failure could be silently swallowed: `XaGroupOpsImpl.commit()` recorded the failure but never surfaced it (`throwIfAnyFailed("commit")` was disabled), and `wrapException()` always wrapped `TransientXaException` inside `JdbcConnectorException`, making the `catch (TransientXaException)` branch in the group-commit loop unreachable. The net effect was that every XA commit failure, transient or permanent, could let a checkpoint be reported successful while the transaction was left prepared-but-uncommitted in the database. This PR restores failure propagation, fixes the retryable/permanent error classification, bounds retries within a single invocation (since Zeta does not persist attempt counters across a checkpoint restart), and reconciles restored checkpoint XIDs against a live resource-manager recovery scan instead of inferring success from an absent XID alone.
# 1. Code Change Review
## 1.1 Core Logic Analysis
Reviewed at head `ca15d97a9ba` (13 commits ahead of `dev` at `43fe63b1fca`, merge-base is the current `dev` tip). The single commit since my last round (`403a0b09f88` -> `ca15d97a9ba`, "[Fix][Connector-V2] Clarify XA restore recovery evidence") is comment-only: it rewords the Javadoc/inline comments on `restoreCommit()` to state that the per-batch recovery-scan refresh is intentional (a DBA or RM cleanup process can resolve an in-doubt transaction between restored batches), addressing li3zhi4's non-blocking Suggestion 3 from 2026-08-21. I confirmed via `git diff 403a0b09f88..ca15d97a9ba` that no executable code changed.
Because this is a full re-review, I retraced the whole runtime path myself rather than relying on my own or others' prior conclusions:
**Failure classification and propagation** (`XaFacadeImplAutoLoad.wrapException`, `XaGroupOpsImpl.commit`): `TRANSIENT_ERR_CODES` is now `{XA_RETRY, XAER_RMFAIL}` (moved off `XA_RBTRANSIENT`, which means the branch was already rolled back — a permanent outcome, not a retryable one). `wrapException` now returns `TransientXaException` directly instead of throwing it wrapped inside `JdbcConnectorException`, so `XaGroupOpsImpl.commit()`'s `catch (XaFacade.TransientXaException e)` branch is reachable, and `result.throwIfAnyFailed("commit")` (re-enabled) now actually fails the checkpoint on a permanent/unknown outcome instead of returning a false success.
**Ordering invariant behind the restore design** (this is the load-bearing correctness property, and I verified it directly rather than taking the PR description's word for it): `XaGroupOpsImpl.commit()` iterates `xids` with the guard `i.hasNext() && (result.hasNoFailures() || allowOutOfOrderCommits)`, and `GroupXaOperationResult.hasNoFailures()` returns false once *either* a permanent failure *or* a transient failure has been recorded. Both production call sites pass `allowOutOfOrderCommits = false`. So within one `commit()` invocation, the loop stops at the *first* failure of any kind, and every subsequent un-iterated XID is appended to `forRetry` untouched, preserving its position. Across `commitXidInfos()`'s own retry rounds, `pending` is rebuilt from `getForRetry()` each round, preserving relative order. This means: a later XID in the list is committed only if every earlier XID already committed successfully. That is exactly the invariant `restoreCommit()` -> `replayRecovered
Checkpoint()` depends on when it treats "everything before the first still-prepared XID" as already resolved. I traced this end to end and it holds on the current head.
**Restore reconciliation** (`JdbcSinkAggregatedCommitter.replayRecoveredCheckpoint`): finds the first checkpoint XID still present in a fresh `xaFacade.recover()` scan (`findFirstRecoveredIndex`); everything before it is logged and skipped as already resolved (justified by the ordering invariant above); everything from that index onward must *all* still be present in the scan or the batch fails closed with a `JdbcConnectorException` before anything in that suffix is committed (no partial commit on the fail-closed path); the still-prepared suffix is then replayed through the same `commitXidInfos` path used by normal commit. An all-absent batch (no checkpoint XID at all in the scan) is treated as already resolved and skipped without ever calling `xaFacade.commit`, which also means restore no longer needs `XAER_NOTA`-as-success inference at all for that case.
This directly resolves the two outstanding `CHANGES_REQUESTED` reviews on this PR:
- **nzw921rx (2026-07-27)** asked for restore to reconcile against `xaFacade.recover()` instead of trusting `XAER_NOTA` alone, with an explicit, durable commit-decision rule. That review predates the recovery-scan reconciliation entirely (it was posted the day after the PR opened, before this mechanism existed). The current head implements exactly the reconciliation shape requested: checkpoint XIDs are intersected with the recovery scan, XIDs not owned by the current batch are never touched (`XidKey`-based canonical comparison, not driver object identity), and unknown/missing XIDs no longer fall back to `XAER_NOTA`-is-success.
- **davidzollo (2026-08-06)** found a concrete non-idempotency bug: if `xidA` commits and `xidB` then permanently fails in the same batch, the success of `xidA` was previously not recorded anywhere durable, so a restart's `restoreCommit()` would re-attempt `xidA`, get `XAER_NOTA` (forgotten, since it was already committed), and treat that as a **new** permanent failure — an infinite restart loop that never even reaches `xidB`. I reconstructed this exact scenario against the current code: `findFirstRecoveredIndex` finds `xidB` (still prepared) at index 1, treats `xidA` (index 0, absent from the scan) as an already-resolved prefix, and only replays `xidB`. `xidA` is never recommitted and never reported as a failure. This is precisely the "commit-order evidence" mechanism the PR description names, and it holds under the ordering invariant traced above.
**li3zhi4 (2026-08-21, COMMENTED, not blocking)** independently re-derived both of the above from source at a slightly earlier head (`61df8fd43`, functionally identical to the current head except for the comment-only commit) and reached the same conclusion: no correctness blocker, three non-blocking suggestions. I re-verified all three against the current head:
- Suggestion 1 (PR description overstates the all-absent case as "fails closed" when the code/docs treat it as already-resolved): still an unaddressed **PR-description-only** discrepancy; code and docs are internally consistent and match what li3zhi4 confirmed. Non-blocking, cosmetic.
- Suggestion 2 (`XaFacade.commit(xid, ignoreUnknown)`'s `ignoreUnknown = true` path has no production caller — both `commitPreparedTransactions` and `replayRecoveredCheckpoint` call it with `false`): confirmed still true on the current head; the `true` path is exercised only by unit tests. Non-blocking, but worth a follow-up: either document the intended production use or narrow the API now that evidence-based replay has replaced `XAER_NOTA`-as-success inference.
- Suggestion 3 (clarify why the recovery scan is refreshed per batch instead of hoisted out of the loop): **addressed** by the one new commit on this head.
## 1.2 Compatibility Impact
Partially incompatible, and correctly disclosed. This is a deliberate, documented behavior change to the XA exactly-once restore path: jobs that previously relied on restore silently treating a missing-with-a-gap XID as success will now fail closed and require operator investigation. `docs/en/introduction/concepts/incompatible-changes.md` and `docs/zh/introduction/concepts/incompatible-changes.md` both carry a new entry describing the affected component, the behavior change, the operational impact, and migration guidance (inspect `XA RECOVER` / `pg_prepared_xacts` for dangling prepared transactions before upgrading). No public API signatures changed in a way that breaks source compatibility; `TransientXaException`'s constructor visibility widened from package-private to `public`, and `XaGroupOps.commit()` gained Javadoc only.
## 1.3 Performance / Side-Effect Analysis
- `restoreCommit()` now performs one `xaFacade.recover()` scan per restored batch instead of zero; this is a bounded, restart-only-path cost, not a steady-state one, and is retried with the same backoff budget as commit (`recoverCheckpointTransactions`) so a transient RM outage during restore does not fail restore immediately.
- A 1-second `Thread.sleep` backoff was added between synchronous retry rounds in both the commit and recovery-scan retry loops. This only fires during genuine XA/RM instability (a real net improvement over the pre-PR behavior, which had effectively zero retries — any non-empty result cancelled the pipeline on the first transient blip). Two of the new unit tests exercise this real sleep (bounded to a couple of seconds each), which is an acceptable, deterministic cost, not a source of flakiness.
- `containsEquivalentXid` is an O(N) linear scan per checkpoint XID against a `Set<XidKey>`, so the reconciliation as a whole is effectively O(N) per batch (the recovered side is already a `Set`). This is restore-path-only and bounded by batch size; not a concern.
## 1.4 Error Handling and Logging
No new formal issues found on this head. All previously raised concerns (nzw921rx, davidzollo) are, on inspection of the current code, structurally resolved rather than merely asserted to be resolved.
# 2. Code Quality Assessment
## 2.1 Coding Standards
New/changed core methods (`restoreCommit`, `replayRecoveredCheckpoint`, `commitXidInfos`, `recoverCheckpointTransactions`, `findFirstRecoveredIndex`, `containsEquivalentXid`, `backoffBeforeRetry`, `wrapException`, `XidKey`) all carry Javadoc explaining intent, constraints, and (where relevant) the fail-closed/already-resolved distinction — this is exactly the kind of "why" documentation this project's `CLAUDE.md` conventions ask for on complex conditional/retry logic, and it is genuinely useful here given how easy this reconciliation logic is to get subtly wrong. No missing-comment gaps found on this pass.
## 2.2 Test Coverage and Test Stability: Stable
- `XaFacadeImplAutoLoadTest` and `XaGroupOpsImplTest` genuinely simulate XA failures by `doThrow`-ing real `XAException` error codes (`XAER_RMFAIL`, `XA_RETRY`, `XA_RBTRANSIENT`, `XAER_NOTA`, `XA_HEURCOM`) from a mocked `XAResource`/`XaFacade`, not just mocking a success path — this was a specific concern going into this round and it checks out.
- `JdbcSinkAggregatedCommitterTest` (8 methods) exercises the reconciliation logic directly: prefix-skip after a still-prepared suffix commits, all-absent-batch skip, fail-closed on a gap after the first recovered XID, transient recovery-scan retry, and bounded-retry exhaustion (`testCommitFailsWhenRetryRoundsNeverDrain` deliberately uses a non-draining `XaGroupOps` stub to prove the outer loop still terminates and fails rather than looping forever — a good defense-in-depth test for a retry loop).
- Notably, this PR also fixes a real, unrelated pre-existing test-infrastructure bug as a side effect: `XaGroupOpsImplIT` (`seatunnel-e2e/.../connector-jdbc-e2e-part-1`) was `@Disabled` with reason "JdbcDatabaseContainer: ClassNotFoundException: com.mysql.jdbc.Driver". This PR removes that `@Disabled` and adds `testCommitFailurePropagatesThroughAggregatedCommitter`, which prepares a real XA transaction against a live MySQL Testcontainer, stops the container to force a genuine RM-unavailable commit failure, and asserts the failure propagates as a `JdbcConnectorException` through the actual `JdbcSinkAggregatedCommitter.commit()` path (not a mock). This is real, non-mocked evidence that the fix works end to end against a real resource manager, which is a meaningfully stronger signal than the unit tests alone.
- All tests are deterministic (mock-driven or single-container-stop), no timing-sensitive assertions beyond the bounded, intentional retry-backoff sleeps noted above.
## 2.3 Documentation Updates
`docs/en/connectors/sink/Jdbc.md` and `docs/zh/connectors/sink/Jdbc.md` both updated to describe the new restore semantics under `max_retries`/`max_commit_attempts`; `docs/en/introduction/concepts/incompatible-changes.md` and the `zh` counterpart both carry the breaking-change entry with migration guidance. Content is consistent with the actual code behavior (verified directly against `replayRecoveredCheckpoint`), not just with the PR description (which, per li3zhi4's Suggestion 1 above, has one stale sentence that overstates the all-absent case — worth a quick edit to the PR description text, no doc or code change needed).
# 3. Architectural Soundness
## 3.1 Elegance of the Solution: Precise fix
The fix is scoped tightly to the XA commit/restore path and does not leak connector-specific logic into `seatunnel-api` or the engine. The commit-order-evidence design is a reasonable way to distinguish "already committed in a prior aborted attempt" from "genuinely lost/unknown" without persisting additional state across checkpoints, which the PR description is upfront about being a constraint (Zeta does not persist attempt counters/partial-success state across a restart today).
## 3.2 Maintainability
Good. The `XidKey` canonical-comparison type, the `commit`/`restoreCommit` split, and the retry/backoff helpers are all small, single-purpose, and documented. The one loose end (li3zhi4 Suggestion 2, `ignoreUnknown=true` with no production caller) is a minor API-clarity nit, not a maintainability risk.
## 3.3 Extensibility
The reconciliation logic operates purely on canonical XID values and the existing `XaFacade`/`XaGroupOps` abstractions, so it should extend cleanly to other XA-capable JDBC dialects without connector-specific branching.
## 3.4 Historical-Version Compatibility
This is the one place where "compatible" needs to be read carefully: this is a **behavior change**, not a wire/state-format change. No checkpoint state schema changed (`XidInfo` serialization is untouched), so upgrading a job with existing checkpointed XA state and then restoring is safe to attempt; the risk is purely operational — a job that was silently succeeding through a masked failure before this fix may now fail closed on restore and require manual investigation, exactly as documented in the incompatible-changes entry. That is the correct trade-off for a data-correctness fix (a checkpoint that silently reports success while data is missing is a worse historical-compatibility problem than a restore that now surfaces the inconsistency), and it is disclosed rather than silent.
# 4. Issue Summary
No formal (numbered) issues found on this head. All prior concerns from other reviewers are addressed in the current code; the two remaining items below are non-blocking carryovers already surfaced by li3zhi4 and not yet actioned:
| Number | Issue | Location | Severity |
| --- | --- | --- | --- |
| N/A-1 | PR description says restore "fails closed" for an all-absent batch; code and docs actually treat it as already-resolved | PR description text only (not code) | Low |
| N/A-2 | `XaFacade.commit(xid, ignoreUnknown=true)` has no production caller after the evidence-based replay design replaced `XAER_NOTA`-as-success inference | `JdbcSinkAggregatedCommitter.java`, `XaFacadeImplAutoLoad.java` | Low |
# 5. Merge Recommendation
### Conclusion: Ready to merge
From a source-correctness standpoint, my independent re-trace of the ordering invariant, the restore reconciliation logic, and the test evidence (including the previously-disabled real-MySQL IT test that now genuinely exercises a live commit failure) finds no blocking issues on the current head (`ca15d97a9ba`). Both outstanding `CHANGES_REQUESTED` reviews were filed against earlier heads and target concerns that the current implementation structurally resolves — nzw921rx's request for recovery-scan-based reconciliation instead of `XAER_NOTA`-as-success inference, and davidzollo's non-idempotent-restart scenario. I want to be precise about what I can and cannot do here: I am not nzw921rx or davidzollo, I cannot dismiss their reviews on their behalf, and `reviewDecision` will correctly keep showing `CHANGES_REQUESTED` until one of them re-reviews the current head or a maintainer with write access revisits and dismisses a stale review. That is a process blocker, not a code blocker,
and it is the only thing standing between this PR and merge as of this comment.
Separately, `mergeStateStatus` is `BLOCKED` and the `Build` check is currently red. I pulled the actual failing job log rather than trusting the status pointer: `updated-modules-integration-test-part-3` failed after 1 hour with `Could not resolve dependencies for project connector-jdbc-e2e-part-2: ... database-commons:1.20.1 ... Connection timed out` — a Maven Central network timeout unrelated to this PR's code, in a module bucket this PR does not touch logically. This is the same class of transient infra failure I flagged and confirmed resolved by rerun in my previous round (that time it was a `jetty-http` 502 from Maven Central); it needs a rerun, not a code change.
1. Blockers (process, not code): the two outstanding `CHANGES_REQUESTED` reviews from nzw921rx and davidzollo need to be revisited against the current head and either re-affirmed with a concrete remaining objection or dismissed/updated — I cannot do this myself as the PR author. CI needs a clean rerun given the confirmed-transient failure cause.
2. Recommended non-blocking fixes: correct the PR description's all-absent-batch wording (li3zhi4 Suggestion 1); consider documenting or removing the unused `ignoreUnknown=true` production path (li3zhi4 Suggestion 2).
No alternative implementation approach looks better here; the commit-order-evidence design is the right shape for this problem given the constraint that Zeta does not persist partial-commit state across a checkpoint restart today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Propagate JDBC XA commit failures [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11569:
URL: https://github.com/apache/seatunnel/pull/11569#issuecomment-5381190204
Addressed JeremyXin's two review points on the new head `0d9f281e054a07190feb664e5130de958268b737`:
1. Added a bounded 1s pause between synchronous XA retry rounds in both `commitXidInfos(...)` and `recoverCheckpointTransactions(...)`, so transient `XAER_RMFAIL` / `XA_RETRY` outages do not burn the whole `max_commit_attempts` budget immediately.
2. Re-enabled `XaGroupOpsImplIT` and added a real-MySQL integration test that prepares an XA transaction, stops the resource manager, and verifies the resulting commit failure now propagates back out through `JdbcSinkAggregatedCommitter.commit(...)` as `JdbcConnectorException` instead of being silently swallowed.
I also updated both `docs/en` and `docs/zh` JDBC sink docs to describe the new synchronous retry pacing. Build is rerunning on this head now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Propagate JDBC XA commit failures [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11569:
URL: https://github.com/apache/seatunnel/pull/11569#issuecomment-5492604274
Since this is my own PR, GitHub won't let me submit a formal review state on it, so this is posted as a plain comment. I re-reviewed the current head from scratch, treating the whole diff and runtime path as if I hadn't written it, rather than skimming the delta.
# What Problem Does This PR Solve?
- User pain point: before this PR, a permanent JDBC XA commit failure could be silently swallowed instead of propagated — `GroupXaOperationResult.throwIfAnyFailed("commit")` was commented out, and `XaFacadeImplAutoLoad.wrapException()` always pre-wrapped `TransientXaException` inside a `JdbcConnectorException`, so `XaGroupOpsImpl.commit()`'s `catch (XaFacade.TransientXaException e)` branch was dead code. A prepared XA transaction could fail to commit and vanish without being retried, reported, or rolled back, while the checkpoint was still reported complete — a real data-loss risk for a two-phase-commit sink.
- Fix approach: restore failure propagation, correct transient-vs-permanent XA error classification, add bounded synchronous retry with backoff, and add a `restoreCommit()` reconciliation path that consults a live `xaFacade.recover()` scan in commit order before replaying restored, still-in-doubt transactions.
- One-sentence summary: this re-review confirms the core JDBC XA fix is unchanged and correct; the only new commit since my last review is an unrelated E2E test-readiness hardening for two other connectors (Azure CosmosDB, Milvus) that happens to concretely fix the two CI failures I flagged as unrelated in my last round.
I re-fetched the head myself (`git log -1` on a fresh clone of `DanielLeens/seatunnel` at `davidzollo_fix_jdbc_xa_commit_failure_20260727`) and confirmed it moved from `18bfbd1743bd` (my last review) to `fbf5b7c99c0d`, one commit: `[Fix][E2E] Wait for Cosmos and Milvus service readiness`. I diffed `18bfbd17..fbf5b7c9` directly and confirmed it touches only `AbstractAzureCosmosDBIT.java` and `MilvusIT.java` — not `XaFacade.java`, `XaFacadeImplAutoLoad.java`, `XaGroupOps.java`, `XaGroupOpsImpl.java`, `GroupXaOperationResult.java`, or `JdbcSinkAggregatedCommitter.java`. I did not stop at the diff stat, though — for a proper re-review I re-read every one of those core files end to end at the current head and independently re-traced the full runtime chain myself (not by re-reading my own prior write-up), documented below.
No new review/comment activity from anyone else landed after my Sept 1 00:30/03:54 rounds at `18bfbd17`, so there's nothing new to acknowledge from other reviewers this round. I did not find anything in this pass that should have been caught in an earlier round but wasn't — my own carried-forward Issue 4 (below) already covers the one non-trivial finding this deep re-trace produced, so no "sorry, missed this" correction is warranted here.
# 1. Code Change Review
## 1.1 Core Logic Analysis
Runtime chain, independently re-traced against the current checked-out head:
```text
notifyCheckpointComplete path (live commit):
CheckpointCoordinator.notifyCompleted()
-> notifyCheckpointCompleted() -> CheckpointFinishedOperation.runInternal()
-> task.notifyCheckpointComplete(checkpointId) SinkAggregatedCommitterTask.java:304-322
-> aggregatedCommitter.commit(aggregatedCommitInfo)
-> JdbcSinkAggregatedCommitter.commit() -> commitPreparedTransactions() -> commitXidInfos()
-> xaGroupOps.commit(pending, false, maxCommitAttempts) XaGroupOpsImpl.java:50-83
-> xaFacade.commit(xid, false) per XID, in order
-> permanent failure: result.failed(x, e)
-> transient failure (XA_RETRY / XAER_RMFAIL only): result.failedTransiently(...)
-> result.throwIfAnyFailed("commit") throws on any permanent failure
-> exception propagates out of notifyCheckpointComplete (declared `throws Exception`, no catch)
-> CompletableFuture.allOf(...).join() in notifyCompleted() throws
-> handleCoordinatorError(..., CHECKPOINT_NOTIFY_COMPLETE_FAILED) CheckpointCoordinator.java:341-357
-> updateStatus(FAILED); checkpointManager.handleCheckpointError(pipelineId, false)
restoreState path (job restart):
SinkAggregatedCommitterTask.restoreState() SinkAggregatedCommitterTask.java:260-282
-> aggregatedCommitter.restoreCommit(aggregatedCommitInfos)
-> per batch: replayRecoveredCheckpoint(checkpointXids, recoverCheckpointTransactions())
-> findFirstRecoveredIndex(): first checkpoint XID still present in a fresh xaFacade.recover() scan
-> everything before that index: treated as already resolved (skip, log WARN)
-> everything from that index onward: must all be present in the scan (fail closed if not) -> commitXidInfos()
```
I did not just re-read my own prior write-up of this chain — I re-derived the checkpoint-failure propagation independently this round by reading `CheckpointFinishedOperation.runInternal()` and `CheckpointCoordinator.notifyCompleted()`/`handleCoordinatorError()` myself, since that's the part of the chain outside the JDBC connector module that actually proves a thrown `JdbcConnectorException` really fails the checkpoint rather than being swallowed somewhere in the engine's operation-retry wrapper. Confirmed: `RetryUtils.retryWithException` in `CheckpointFinishedOperation` only retries `TaskGroupContextNotFoundException`, so our exception is wrapped once in `SeaTunnelEngineException` and rethrown, and `notifyCompleted()`'s `catch (Throwable e)` routes it to `handleCoordinatorError`, which marks the checkpoint coordinator `FAILED` and triggers `checkpointManager.handleCheckpointError`. This is the fix actually working end-to-end, not just failing loudly inside the connector and gett
ing lost above it.
**One finding worth stating precisely (carried forward as Issue 1 below, not new):** `XaFacade.commit(xid, ignoreUnknown)` is hardcoded to `false` at its only production call site (`XaGroupOpsImpl.java:61`), so `XAER_NOTA` ("Xid is not valid to RM") is never in `TRANSIENT_ERR_CODES` and always surfaces as a permanent failure. I traced the concrete scenario this matters for: if an external actor (a DBA, or RM in-doubt-transaction cleanup tooling) resolves a checkpoint's XID between `recover()` and the following `xaFacade.commit()` call for the same batch — the residual gap `replayRecoveredCheckpoint()`'s Javadoc explicitly documents as narrowed-but-not-eliminated — the live commit or restore call fails hard instead of silently treating the already-resolved XID as done. I traced what happens next: the resulting job restart re-enters `restoreState()`/`restoreCommit()`, which re-scans and this time correctly finds that XID absent-and-skippable (since it's now genuinely resolved),
so the batch converges within one extra restart cycle rather than looping forever. This is a self-healing, operator-visible "spurious restart," not a livelock and not silent data loss — and it is the intentional, safer trade-off versus the alternative (treating any `XAER_NOTA` as "fine, already resolved," which would also mask a genuinely corrupted/fabricated Xid). This exact race and trade-off is already discussed at length by `@dybyte` and me in the `JdbcSinkAggregatedCommitter.java:105` review thread (2026-08-27), and the `restoreCommit()` Javadoc was tightened in `c5f4ec0a0d` to name the external-actor scenario explicitly — so this is confirmation of already-surfaced, already-addressed-in-docs behavior, not a new gap.
## 1.2 Compatibility Impact
**Partially incompatible, and disclosed** — unchanged from prior rounds. No wire/state serialization format change. The observable behavior change (a permanent XA commit failure now fails the checkpoint instead of being silently absorbed, and a missing XID after a still-prepared one fails restore closed instead of assuming success) is the intended fix and is documented in `docs/en/introduction/concepts/incompatible-changes.md` / `docs/zh/...` under "JDBC XA restore now uses recovery-order evidence and fail-closed gaps." I re-read the current doc text against the current code myself this round rather than trusting the prior summary, and it accurately matches `replayRecoveredCheckpoint()`'s actual behavior (prefix-skip, suffix-fail-closed, all-absent-skip).
## 1.3 Performance / Side-Effect Analysis
Unchanged from prior rounds: bounded retry (`maxCommitAttempts`, default 3, 1s fixed backoff via `Thread.sleep` in `backoffBeforeRetry`), executed on the aggregated committer's own dedicated task thread (not a shared engine thread pool), triggered only at checkpoint-complete/restore boundaries, not per-record. No unbounded collections — `pending`/`toRetry`/`stillPrepared` are all bounded by the batch's own XID count.
## 1.4 Error Handling and Logging
No swallowed exceptions on the paths I traced. Every hard-failure path throws a `JdbcConnectorException` naming the affected XIDs (or wraps a `TransientXaException`/`XAException` with its error code), and log levels are appropriate (`WARN` for skip/already-resolved/retry-exhaustion-adjacent conditions, `ERROR`-equivalent via the thrown exception for real failures). No credentials or connection strings appear in any of the touched log statements.
## 2. Code Quality Assessment
## 2.1 Coding Standards
Javadoc present on all new/changed non-trivial methods (`restoreCommit`, `replayRecoveredCheckpoint`, `recoverCheckpointTransactions`, `findFirstRecoveredIndex`, `containsEquivalentXid`, `backoffBeforeRetry`), ASF license headers present, no wildcard imports in any touched file.
## 2.2 Test Coverage and Test Stability
**Core JDBC/XA logic: Stable.** Unit coverage (`XaFacadeImplAutoLoadTest`, `XaGroupOpsImplTest`, `JdbcSinkAggregatedCommitterTest`) exercises every reconciliation branch (recovered-by-value matching, unrelated recovered XIDs, already-resolved prefixes, no-evidence missing gaps failing closed, bounded transient-retry for both commit and the recovery scan). `XaGroupOpsImplIT` runs against a real MySQL testcontainer, not a mock, and its `testCommitFailurePropagatesThroughAggregatedCommitter` drives an actual XA prepare -> forced commit failure -> propagation through the real committer.
I independently verified the current CI run for this exact head (`fbf5b7c99c0d`, fork run `33476561911`) rather than trusting the prior round's numbers: 78 success, 11 skipped, 2 still in progress at review time, 2 failures — `edge-agent-it (8)` and `doris-connector-it (8)`, both clearly unrelated to this diff's connector/module. Every `jdbc-connectors-it-part-*` (1-7, both JDKs), `jdbc-connectors-it-ddl` (both JDKs), and `unit-test` (all four OS/JDK combinations) job is green. Notably, `all-connectors-it-4` and `all-connectors-it-7` — the two jobs I diagnosed as pre-existing/unrelated flakes in my last round (`AzureCosmosDBSourceIT` and `MilvusIT` failures) — are now green on both JDKs in this run, which is concrete evidence that this round's new commit (the Cosmos/Milvus readiness-wait hardening) actually fixed those two flakes rather than just being incidental scope creep.
**Stability rating for the new commit's own test-infra change: Stable.** `seedContainerWhenReady`/`waitForMilvusReady` use `Awaitility.await().ignoreExceptions().pollInterval(...).atMost(...).untilAsserted(...)` — a real readiness assertion (retrying the actual seed/RPC call until it succeeds), not a bare fixed sleep. `MilvusIT` previously used a bare `Awaitility.given().ignoreExceptions().await().atMost(720L, SECONDS)` with no assertion inside `await()` at all (that call by itself waits the fixed duration unconditionally since nothing is being asserted) — replacing it with a real `listDatabases()` RPC-backed readiness check is a strict improvement, not a loosened tolerance: it now fails fast if the real precondition isn't met instead of just sleeping a fixed window and hoping.
## 2.3 Documentation Updates
Unchanged from prior rounds — `docs/en(zh)/connectors/sink/Jdbc.md` and `docs/en(zh)/introduction/concepts/incompatible-changes.md` are consistent with the current code, verified again this round (see 1.2).
# 3. Architectural Soundness
## 3.1 Elegance of the Solution
Precise fix: restores previously-disabled failure propagation, corrects a specific error-code misclassification (`XA_RBTRANSIENT` — already permanently rolled back — replaced with `XA_RETRY` — call had no effect, safe to reissue — in `TRANSIENT_ERR_CODES`), and adds order-preserving, evidence-based restore reconciliation rather than a new persisted-state mechanism.
## 3.2 Maintainability
Good — the reconciliation invariant (`XaGroupOpsImpl.commit()` only ever attempts XIDs in original order with `allowOutOfOrderCommits=false`, so a still-prepared XID at index *k* implies everything after *k* in the same batch must also still be prepared) is now explicit in both code comments and the incompatible-changes doc, not left implicit.
## 3.3 Extensibility
No concerns raised this round.
## 3.4 Historical-Version Compatibility
No checkpoint/savepoint state-schema change — `XidInfo`'s serialized shape is untouched, only the in-memory reconciliation logic around it changed. A job with in-flight XA-prepared transactions from before this upgrade restores exactly the same way as one prepared after it.
# 4. Issue Summary
| Number | Issue | Location | Severity | Raised by another reviewer |
| --- | --- | --- | --- | --- |
| 1 (carried) | `XaFacade.commit(xid, ignoreUnknown=true)` still has no production caller; `ignoreUnknown` is hardcoded `false` at the only call site, so an externally-resolved XID (DBA/RM cleanup) always surfaces as a hard failure rather than being tolerated. Self-healing within one restart cycle (traced in 1.1), and the trade-off is intentional, but the unused `true` branch is dead code today. | `JdbcSinkAggregatedCommitter.java` (call site) / `XaFacadeImplAutoLoad.java` (`buildCommitErrorDesc`) | Low | Yes — discussed at length with @dybyte (2026-08-27); Javadoc hardened in `c5f4ec0a0d` |
| 2 (carried) | PR description text previously described the all-absent case as "fails closed," while code/docs correctly treat it as "already resolved" | PR description | Low | No (self-carried) |
No High or Medium issues found this round or carried from before.
# 5. Merge Recommendation
### Conclusion: Ready to merge after fixes
1. **Blockers — must be fixed (process, not code; I cannot act on these myself as the PR author):**
- `reviewDecision` is still `CHANGES_REQUESTED`, driven by two still-open, unresolved formal reviews: `@nzw921rx` (2026-07-27, at commit `f836f8d6`) and `@davidzollo` (2026-08-06, at commit `8bb49c06`). Both predate the current recovery-scan/commit-order reconciliation design by many rounds. Neither reviewer has formally dismissed or updated their review state on GitHub as of this comment. This needs a maintainer with write access to re-review and clear the stale gate, or for `@nzw921rx`/`@davidzollo` to update their own review state — I cannot dismiss another reviewer's formal review, and being the PR author I cannot submit my own formal review state at all (hence this plain comment).
- CI on the current head still has 2 in-progress jobs at review time; confirm the run finishes with no new failures beyond the two already-diagnosed-unrelated ones (`edge-agent-it`, `doris-connector-it`).
2. **Recommended fixes — non-blocking:**
- Issue 1 (Low): remove the dead `ignoreUnknown=true` capability, or leave it as documented, deliberate defense against `XAER_NOTA` being conflated with "this transaction was legitimately externally resolved" — either is defensible, but the current unused-branch state should be a conscious choice, not an oversight.
- Issue 2 (Low): tighten the PR description's own wording to match the code/docs ("treated as already resolved," not "fails closed") for the all-absent-batch case.
**Overall assessment:** this re-review found the core fix unchanged, correct, and — where I dug deeper than a diff-stat check (the checkpoint-failure propagation chain outside the connector module, and the `XAER_NOTA`/external-actor race) — independently re-derivable from the source rather than merely plausible. The one new commit this round is a well-targeted, unrelated E2E test-readiness hardening that concretely fixed the two flaky CI jobs I'd previously diagnosed as unrelated noise. The two open blockers remain purely procedural: stale `CHANGES_REQUESTED` states from reviews that predate the current design, and CI finishing. A different maintainer needs to action the review-gate item; I can't do that on my own PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Propagate JDBC XA commit failures [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11569:
URL: https://github.com/apache/seatunnel/pull/11569#issuecomment-5461958108
Note: this PR is authored by me (DanielLeens), so GitHub blocks a self-review submission. Posting this as a plain issue comment instead of a formal PR review, per project convention for self-authored PRs (same as my previous rounds on this thread).
New activity since my last full re-review (comment above, at head `ca15d97a9bad`): one new commit, `a9257e75bf1a` ("[Chore] Retry CI"), and a new `APPROVED` review from @dybyte on the prior head. I re-ran the analysis below from scratch against the current head rather than assuming the prior round still applies.
# What Problem Does This PR Solve?
Unchanged from my last full re-review: before this patch, a permanent JDBC XA commit failure could be silently swallowed (`XaGroupOpsImpl.commit()` recorded the failure but `throwIfAnyFailed("commit")` was disabled, and `wrapException()` always wrapped `TransientXaException` so the retryable-error branch was unreachable). This PR restores failure propagation, fixes retryable/permanent error classification, bounds retries within a single invocation, and reconciles restored checkpoint XIDs against a live `xaFacade.recover()` scan instead of inferring success from an absent XID alone.
# 1. Code Change Review
## 1.1 Core Logic Analysis
I diffed the current head against the exact commit I fully re-reviewed less than a day ago:
```
git diff ca15d97a9bad a9257e75bf1a9fb63bcdaca3130baa3c9c585990
```
This is empty. `a9257e75bf1a` is a genuine zero-diff commit (verified via `git show --stat`, no files listed) authored by @davidzollo purely to retrigger CI — not a rebase, not a squash-with-hidden-change, an actual empty commit on top of the same tree. So the full from-scratch analysis I already posted for `ca15d97a9bad` (ordering invariant in `XaGroupOpsImpl.commit()`, the `findFirstRecoveredIndex`/`replayRecoveredCheckpoint` reconciliation, the nzw921rx and davidzollo scenarios, li3zhi4's three suggestions) stands unchanged on the current head byte-for-byte — I re-verified this via the diff above rather than assuming it. I'm not going to re-paste that ~2000-word analysis verbatim here since it would add no new information; it's the comment immediately above this one on the same head-equivalent code.
What actually changed this round is CI, addressed in full below.
## 1.2 Compatibility Impact
Unchanged: partially incompatible, correctly disclosed in `docs/en(zh)/introduction/concepts/incompatible-changes.md` — jobs relying on the old silent-success-on-gap restore behavior will now fail closed and need operator investigation.
## 1.3 Performance / Side-Effect Analysis
Unchanged: bounded, restart-only-path `recover()` scan cost; 1s backoff between synchronous retry rounds only fires under genuine XA/RM instability.
## 1.4 Error Handling and Logging
No new issues. All previously raised concerns (nzw921rx, davidzollo) remain structurally resolved on this head, since the code is identical to what I already verified line-by-line.
**CI diagnosis for the current head (`a9257e75bf1a`)** — this is the new, real work this round. `Build` is red on the apache-side status, which is just a pointer; the real run is on the fork. I pulled every failing job's actual log rather than trusting the red X:
- `Run / unit-test (8, ubuntu-latest)` — `CoordinatorServiceTest.testClearCoordinatorServiceDropsPendingJobsUnderRejectStrategy:642 » ConditionTimeout` (Awaitility timeout waiting on job-scheduling state in `seatunnel-engine-server`). This PR's diff (verified via `git diff <merge-base 43fe63b1fca> a9257e75bf1a` — the correct comparison, since `dev` has moved ~25 commits past this PR's merge-base and diffing against current `dev` tip pulls in unrelated noise) touches only `connector-jdbc`'s XA classes, its own tests, `connector-jdbc-e2e-part-1`, and docs. It does not touch `seatunnel-engine-server`, `CoordinatorService`, or anything in its call path. I also checked: no commit between this PR's merge-base and current `dev` touched `CoordinatorServiceTest.java` or `CoordinatorService.java`, so this isn't a case of a known-fixed flake regressing — it's an Awaitility-based engine unit test racing against Hazelcast job-scheduling state, in a module this PR never touches.
- `Run / updated-modules-integration-test-part-3 (8, ubuntu-latest)` — `connector-jdbc-e2e-part-2` (not this PR's own `connector-jdbc-e2e-part-1`) failed at the Maven dependency-resolution stage: `Could not transfer artifact com.google.code.gson:gson:pom:2.13.1 ... Connection reset`, a transitive dependency of `connector-milvus`. Pure Maven Central network flake, same class of failure (different artifact) as the `database-commons` timeout I diagnosed on the previous head.
- `Run / updated-modules-integration-test-part-2 (8, ubuntu-latest)` and `(11, ubuntu-latest)` — both failed on the exact same test, `PostgresCDCIT.testPostgresCdcSnapshotOnlyAndCommittedOffsetStartupModes:614 » ConditionTimeout` (`expected: <1> but was: <0> within 3 minutes`), on two different JDKs in the same run. `PostgresCDCIT` belongs to `connector-cdc-postgres`, which this PR does not touch at all. The identical failure signature across two independent JDK runs is itself evidence this is a pre-existing environmental/timing flake in the CDC container setup, not something a code change in this run introduced (there is no code change in this run).
None of the four failures touch a file, module, or call path this PR's diff modifies. This needs a CI rerun, not a code change — consistent with the transient-infra pattern from my previous two CI triage rounds on this same PR.
# 2. Code Quality Assessment
## 2.1 Coding Standards
Unchanged: all new/changed core methods carry Javadoc; no gaps found on the previous from-scratch pass, and there's no code delta to re-check.
## 2.2 Test Coverage and Test Stability: Stable
Unchanged rating. No test files changed in this commit. The genuinely new signal this round is negative-but-irrelevant: none of the newly-failing CI tests belong to this PR's own test suite (`XaFacadeImplAutoLoadTest`, `XaGroupOpsImplTest`, `JdbcSinkAggregatedCommitterTest`, `XaGroupOpsImplIT`) — those all still pass in every failing job's log; only unrelated modules failed.
## 2.3 Documentation Updates
Unchanged: `docs/en(zh)/connectors/sink/Jdbc.md` and the incompatible-changes docs are current and match the code; no doc delta in this commit.
# 3. Architectural Soundness
## 3.1 Elegance of the Solution: Precise fix
Unchanged.
## 3.2 Maintainability
Unchanged.
## 3.3 Extensibility
Unchanged.
## 3.4 Historical-Version Compatibility
Unchanged: no checkpoint state schema changed; the risk is purely operational and disclosed.
# 4. Issue Summary
No new code-level issues on this head (no code changed). Carrying forward the two non-blocking items from my previous round, unrenumbered:
| Number | Issue | Location | Severity |
| --- | --- | --- | --- |
| N/A-1 | PR description says restore "fails closed" for an all-absent batch; code and docs actually treat it as already-resolved | PR description text only (not code) | Low |
| N/A-2 | `XaFacade.commit(xid, ignoreUnknown=true)` has no production caller after the evidence-based replay design replaced `XAER_NOTA`-as-success inference | `JdbcSinkAggregatedCommitter.java`, `XaFacadeImplAutoLoad.java` | Low |
# 5. Merge Recommendation
### Conclusion: Ready to merge
The code on the current head is byte-for-byte identical to what I already fully re-traced (ordering invariant, restore reconciliation, both outstanding `CHANGES_REQUESTED` scenarios, li3zhi4's suggestions) in my previous comment — I verified that identity directly via diff rather than assuming it, so this is a genuine re-review conclusion, not a copy-paste. @dybyte's `APPROVED` review on the prior head is further independent confirmation.
What's actually new this round is the CI failure, and I traced it to four jobs, none of which touch a file this PR's diff modifies: an unrelated engine unit test (`CoordinatorServiceTest`), a Maven Central network reset resolving an unrelated module's dependency (`connector-jdbc-e2e-part-2`/`gson`), and the same pre-existing `PostgresCDCIT` Awaitility timeout on two JDKs. This is infra flake, not a regression from this commit (which changed no files) or from this PR's diff (which never touches any of the three affected modules).
1. Blockers (process, not code, same as last round): the two outstanding `CHANGES_REQUESTED` reviews from @nzw921rx and @davidzollo need to be revisited against the current head and re-affirmed or dismissed by them or a maintainer with write access — I cannot do this myself as the PR author. CI needs a clean rerun given the four confirmed-unrelated failure causes above.
2. Recommended non-blocking fixes: same two carryovers as before (PR description wording, unused `ignoreUnknown=true` path).
No alternative implementation approach changes here; nothing in this round's findings affects the design.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Propagate JDBC XA commit failures [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11569:
URL: https://github.com/apache/seatunnel/pull/11569#issuecomment-5342836639
This is a re-verification of the current head, `fa400efc26840cf2ae12eef437c7c6c4bd55cd2d`, one commit past what I reviewed a few hours ago (`17234b67e1d5`, itself a no-op CI retrigger on top of `03f7fd4347f7`). This new commit, `[Fix][Connector-V2] Treat fully absent XA restore batches as resolved`, is a direct response to Issue 1 from my last review. I re-traced it from scratch against the actual source rather than assuming the fix is correct because the commit title matches what I asked for.
# What Problem Does This PR Solve?
- User pain point: in the JDBC XA (2PC) aggregated-commit path, a permanent XA commit failure could be silently swallowed (`throwIfAnyFailed("commit")` was commented out, and `TransientXaException` was double-wrapped so its catch clause was dead code). A checkpoint could report success while a transaction stayed prepared-but-uncommitted in the database.
- Fix approach: reclassify `XA_RETRY`/`XAER_RMFAIL` as transient (not `XA_RBTRANSIENT`, which is a permanent already-rolled-back outcome), stop swallowing permanent commit failures, and reconcile checkpoint-restored XIDs against a fresh `xaFacade.recover()` scan using commit order as evidence instead of a blanket `XAER_NOTA` tolerance.
- One-sentence summary: this turns a silent "reports success, loses the commit" bug into a fail-closed commit/restore path, and this latest commit fixes the one case where "fail-closed" had itself become a permanent availability regression (a fully-committed batch surviving an unrelated restart).
# 1. Code Change Review
## 1.1 Core Logic Analysis
The changed method is `JdbcSinkAggregatedCommitter.replayRecoveredCheckpoint` (`seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/sink/JdbcSinkAggregatedCommitter.java:194-226`), called from `restoreCommit` once per `JdbcAggregatedCommitInfo` on job/task restore.
Before (what I flagged as Issue 1):
```java
int firstRecoveredIndex = findFirstRecoveredIndex(checkpointXids, recoveredXids);
if (firstRecoveredIndex < 0) {
throw new JdbcConnectorException(
CommonErrorCodeDeprecated.WRITER_OPERATION_FAILED,
String.format(
"none of the restored checkpoint transactions are present in the XA recovery scan: %s",
checkpointXids));
}
```
After:
```java
int firstRecoveredIndex = findFirstRecoveredIndex(checkpointXids, recoveredXids);
if (firstRecoveredIndex < 0) {
log.warn(
"Skipping checkpoint batch because none of its transactions remain in the XA recovery scan; treating it as already resolved: {}",
checkpointXids);
return;
}
```
The rest of the method — the "gap after a still-prepared XID fails closed" branch at `:206-217` — is untouched.
Key findings:
- Normal path reached: yes. `restoreCommit` is on Zeta's task-restore critical path — it runs whenever a JDBC XA sink task restarts from a checkpoint, which includes both real failure recovery and any unrelated restart (source/task failure elsewhere, node loss) that happens to fall in the window between a checkpoint's XA commit completing and the next checkpoint independently completing.
- Scenario covered: the specific scenario I raised as Issue 1 — `CheckpointCoordinator` advances `latestCompletedCheckpoint` before phase-2 XA commit for that checkpoint even runs, and holds it there until the next checkpoint completes on its own. Any restart in that window replays `restoreCommit()` against a `XidInfo` batch that a prior, healthy `notifyCheckpointComplete()` already fully committed. Every XID in that batch is then legitimately absent from the RM (committed and forgotten — standard XA behavior), so the old code's unconditional throw converted the single most common healthy-restart case into a permanent failure loop with no operator remedy (the RM state doesn't change between retries).
- Precise fix, not a workaround: it targets exactly the boundary condition I identified — total absence, where the model has zero still-prepared evidence to anchor the prefix/suffix invariant against — and leaves the rest of the commit-order reconciliation (which correctly stayed fail-closed for the "gap after a still-prepared XID" case, since that pattern cannot arise from a legitimate sequential commit) completely alone. One branch, one behavior change.
- Remaining hole (not new, inherent to XA, already priced into my prior review): treating total absence as "already resolved" is still a heuristic. XAER_NOTA-equivalent absence at scan time cannot distinguish "committed and forgotten" (the common, targeted case) from "rolled back/expired/heuristically completed for an unrelated reason and then discarded by the RM." A fully rigorous fix needs an external durable record of "we ourselves issued a successful commit," independent of RM bookkeeping — out of scope for a one-branch fix. Given the WARN log at the point of tolerance and that this narrows rather than widens the pre-existing ambiguity (the pre-PR code tolerated absence unconditionally in every case, not just this one), I'm not re-opening this as a new blocker; it's the same class of residual risk every version of this PR has carried, not something this commit makes worse.
- Runtime path (restore, current head):
```text
Task restart / restore
-> JdbcSinkAggregatedCommitter.restoreCommit(aggregatedCommitInfos) [L91-99]
-> tryOpen()
-> for each JdbcAggregatedCommitInfo:
-> recoverCheckpointTransactions() [L234-255]
-> xaFacade.recover(), bounded retry on TransientXaException
-> replayRecoveredCheckpoint(checkpointXids, recoveredXids) [L194-226]
-> findFirstRecoveredIndex() [L264-271]
-> branch: no checkpoint XID found in scan (firstRecoveredIndex < 0)
-> log.warn(...), return [FIXED: was throw]
-> branch: some XID still prepared
-> verify every XID from firstRecoveredIndex onward is present
-> absent -> throw (unchanged, fail-closed, correct)
-> commitXidInfos(stillPrepared, ignoreUnknown=false) [L218]
-> log.warn for the already-resolved prefix, if any [L219-225]
```
## 1.2 Compatibility Impact
**Classification: Fully compatible.** No API/config/serialization change — `XidInfo`/`JdbcAggregatedCommitInfo` untouched, no new/renamed config option. The behavioral change is a narrowing of the fail-closed surface introduced earlier in this same PR (not yet released), so there's no compatibility break relative to any shipped version. `docs/en` and `docs/zh` (`Jdbc.md` and `incompatible-changes.md`) were updated in the same commit and now accurately describe "all-absent -> treated as already resolved, skip" vs. "gap after a still-prepared XID -> fail closed" — I checked both language versions and they say the same thing.
## 1.3 Performance / Side-Effect Analysis
No change in this commit. Carryover from my last pass, still true: `recoverCheckpointTransactions()` runs once per `JdbcAggregatedCommitInfo` inside `restoreCommit`'s loop rather than once per invocation — a pre-existing minor inefficiency (extra RM round-trips when a restore batch has multiple `JdbcAggregatedCommitInfo` entries), not a correctness issue, and not something this commit touches. `commitXidInfos`'s round cap still has no backoff between rounds (Issue 2 below).
## 1.4 Error Handling and Logging
The fix moves this branch from "throw, opaque to a human reading the checkpoint failure without XA context" to "WARN with the full XID list, then continue" — this is strictly better for both correctness (no more permanent wedge) and operability (the WARN gives an operator something to grep for if they want to double check RM history). I verified `XaFacadeImplAutoLoad`'s error classification independently (not just re-reading the prior review): `TRANSIENT_ERR_CODES = {XA_RETRY, XAER_RMFAIL}` (`XaFacadeImplAutoLoad.java:75-76`) correctly excludes `XA_RBTRANSIENT` (a permanent already-rolled-back outcome per the XA spec, not a "no effect, retry me" outcome), and `buildCommitErrorDesc` (`:323-331`) only tolerates `XAER_NOTA` when `ignoreUnknown=true` — every other code, including the `XA_HEURRB`/rollback family, still propagates as a hard failure. This confirms the classification correctness I'd only taken on trust before.
**Issue 2 (Medium, carryover, unchanged by this commit):** no backoff between synchronous commit/recover retry rounds — `JdbcSinkAggregatedCommitter.java:129-142` (`commitXidInfos`) and `:237-248` (`recoverCheckpointTransactions`). `XAER_RMFAIL`-class outages will burn the whole `max_commit_attempts` budget in microseconds with no delay between attempts.
# 2. Code Quality Assessment
## 2.1 Coding Standards
The updated Javadoc on `restoreCommit` (`:84-89`) matches the new behavior ("An all-absent batch is treated as already resolved..."). No missing documentation on the changed method.
## 2.2 Test Coverage and Test Stability
`JdbcSinkAggregatedCommitterTest.testRestoreCommitSkipsAlreadyResolvedBatchWhenRecoveryScanHasNoCheckpointXid` (`:143-163`) was rewritten in the same commit: it now uses a 2-XID all-absent batch, asserts `assertDoesNotThrow`, and verifies `xaGroupOps.commit(...)` is `never()` invoked — which matches the implementation exactly (the method returns before building the `stillPrepared` list, so commit is genuinely never attempted, not just swallowed). This closes the exact gap I called Issue 3 last time ("no test where the entire batch is legitimately already-committed") at the unit level.
**Stability rating: Stable.** Mockito-based, single-threaded, deterministic on object identity/counts/message content; no `Thread.sleep`, no shared static state, no `@DisplayName`.
**Issue 3 (Medium, carryover, partially addressed):** the unit-level gap is now closed, but there is still no integration/real-database exercise of any part of this PR's XA commit/restore path — `XaGroupOpsImplIT` (`seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-part-1/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/xa/XaGroupOpsImplIT.java`) remains `@Disabled` and untouched by this PR (confirmed: `git diff dev...HEAD` for that path is empty).
## 2.3 Documentation Updates
`docs/en/connectors/sink/Jdbc.md`, `docs/zh/connectors/sink/Jdbc.md`, and both `incompatible-changes.md` files were updated in this same commit to describe the corrected all-absent behavior. Consistent between languages; I read both.
# 3. Architectural Soundness
## 3.1 Elegance
Precise fix. One branch changed from throw to warn-and-return, with an updated regression test and updated docs in the same commit — no redesign, no scope creep into the (still correctly fail-closed) gap-after-recovered branch.
## 3.2 Maintainability
Unchanged from my last assessment: the responsibility split across `XaFacadeImplAutoLoad` (driver-outcome classification), `XaGroupOpsImpl` (grouped commit propagation), and `JdbcSinkAggregatedCommitter` (restore reconciliation) stays clean; `replayRecoveredCheckpoint`/`findFirstRecoveredIndex`/`recoverCheckpointTransactions` remain small and single-purpose.
## 3.3 Extensibility
Unchanged: `XidKey`'s canonical `formatId`+GTRID+BQUAL comparison is a solid, reusable primitive; `XaGroupOps.commit(...)`'s new `default` overload lets other implementors opt in without a forced signature break.
## 3.4 Historical-Version Compatibility
No checkpoint-state schema change; old checkpoints/savepoints deserialize unmodified. This fix is a decision-logic change, not a serialization/format change, and applies uniformly regardless of checkpoint age.
# 4. Issue Summary
| # | Issue | Location | Severity | Status |
|---|-------|----------|----------|--------|
| 1 | ~~All-absent restore batch threw unconditionally, misclassifying "already fully committed by a prior healthy checkpoint, unrelated restart" as unrecoverable, wedging the job permanently~~ | `JdbcSinkAggregatedCommitter.java:198-205` | High | **Fixed in `fa400efc2`, verified** |
| 2 | No backoff between synchronous commit/recover retry rounds; `max_commit_attempts` is largely inert against `XAER_RMFAIL`-class outages | `JdbcSinkAggregatedCommitter.java:129-142`, `:237-248` | Medium | Open, carryover |
| 3 | No end-to-end/real-database test exercises commit/restore failure propagating to checkpoint/job failure; `XaGroupOpsImplIT` remains `@Disabled` and untouched by this PR | `XaGroupOpsImplIT.java` (connector-jdbc-e2e-part-1) | Medium | Open, carryover (unit-level gap for the specific all-absent scenario is now closed) |
| 4 | Fork `Build` run for the current head (`fa400efc2`) was still in progress at review time: all completed jobs green, including every `unit-test` lane (which runs this PR's own new/changed tests), zero failures observed across 56 completed jobs; ~15 `updated-modules-integration-test-part-*` jobs still running | fork run `32248144048` (in progress) | Low (process, not source) | Not yet finalized |
# 5. Merge Recommendation
### Conclusion: Ready to merge after fixes
1. **Blockers — must be fixed**
- None from source review. Issue 1, the sole High/blocking finding from my prior pass, is fixed and independently re-verified against the current head. The only open item before this can be called done is Issue 4: let the in-progress fork CI run for `fa400efc2` finish and confirm it matches the all-green result already observed for every completed job (especially the `unit-test` lanes, which already cover this exact fix on this exact SHA).
2. **Recommended fixes — non-blocking**
- Issue 2 (Medium): add bounded backoff between synchronous retry rounds.
- Issue 3 (Medium): re-enable `XaGroupOpsImplIT` with real XA coverage, including the "whole batch already committed by a prior successful commit, then an unrelated restart" scenario at the integration level (the unit test now covers this at the mock level).
**On the reconciliation logic overall:** I re-confirm what I found in my morning pass — the commit-order/prefix-suffix invariant correctly closes both @nzw921rx's 2026-07-27 normalization concern and @davidzollo's 2026-08-06 idempotency concern, and this commit closes the one new problem I found on top of that (Issue 1). No new problem introduced by this commit; the change is narrowly scoped to exactly the branch I asked about, with a matching regression test and matching doc update in the same commit.
**Overall assessment:** the engineering direction has been sound throughout this PR's review history, and every round has converged rather than regressed. This round closes the last blocking item I had open. Once the current fork CI run finishes green, I'd expect to be able to call this ready to merge without qualification.
Since this is my own PR, this is posted as a comment rather than an approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Propagate JDBC XA commit failures [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11569:
URL: https://github.com/apache/seatunnel/pull/11569#issuecomment-5552990238
Self-review CI note (own PR): the retrigger run (`ba3c92216`, still in progress) already shows `updated-modules-integration-test-part-2 (8)` red — but before `XaGroupOpsImplIT` ran: Maven could not resolve `org.testcontainers:postgresql:jar:1.17.6` for `connector-jdbc-e2e-part-1` (Central transfer failure of the artifact descriptor), the preceding Hudi ITs in the same job passed. Pure infra, third distinct Maven Central/registry blip on this PR's runs, no XA-logic signal. Will retrigger once the run completes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Propagate JDBC XA commit failures [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11569:
URL: https://github.com/apache/seatunnel/pull/11569#issuecomment-5565034464
Thanks @DanielLeens for tracing the `b56f423a1` build — agreed, the `FlussSinkIT` timeout is unrelated to this diff, which only touches `connector-jdbc`'s XA path and `XaGroupOpsImplIT`. Retriggering just that job sounds right.
While CI reruns, the earlier review points are still open. Could you please address (or push back on) each of these:
1. **Empty recovery scan in `restoreCommit()`**: an all-absent XA RECOVER result is currently treated as "already resolved", so a truncated or incomplete scan would silently drop prepared data. Please distinguish "scan returned nothing" from "each xid was positively confirmed resolved", or fail loudly in the former case.
2. **Re-enabled commit failure propagation**: the new reconciliation only lives in `restoreCommit()`, but engines such as Flink/Spark replay restored commit infos through `commit()`. Please either apply the same reconciliation on the `commit()` path or explain how restore works on those engines.
3. **Prefix-skip logic**: "absent from XA RECOVER" is equated with "committed", so an externally rolled-back transaction is accepted as success. Please clarify how a rollback is detected, or document the limitation.
4. **Fixed 1s backoff × retry budget on the committer thread**: this can block the checkpoint/commit thread long enough to trip checkpoint timeouts, the recovery scan is re-run per batch, and the synchronous sleep needs explicit interruption handling. Please make the backoff configurable/bounded, reuse the scan across a batch where possible, and handle interruption properly.
5. **`TRANSIENT_ERR_CODES` reclassification** (XA_RBTRANSIENT → permanent, XA_RETRY → retryable) plus the unwrapped `TransientXaException`: this applies to every XA verb going through the facade, not just commit, and changes the connector error-code contract on surfaced failures. Please scope it to commit or justify the global change, and keep surfaced failures wrapped in the connector exception type.
6. **Docs**: since the error-code reclassification is user-facing, please add an entry to the incompatible-changes docs.
Once those are addressed and the retriggered job finishes, I'll take another pass.
<!-- streview-comment:872 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Propagate JDBC XA commit failures [seatunnel]
Posted by "JeremyXin (via GitHub)" <gi...@apache.org>.
JeremyXin commented on PR #11569:
URL: https://github.com/apache/seatunnel/pull/11569#issuecomment-5371684900
Reviewed the latest head 61df8fd43. The changes look good overall — found two issues worth confirming before merge:
**Issue 1**: No backoff between synchronous XA commit/recover retry rounds (MAJOR)
Location: JdbcSinkAggregatedCommitter.java — commitXidInfos (L129-142) / recoverCheckpointTransactions (L237-248)
Description: Both bounded retry loops are tight synchronous loops with no delay between rounds. For XAER_RMFAIL-class outages (resource manager unavailable), the whole max_commit_attempts budget is consumed in microseconds, making the retry mechanism effectively inert for the exact failure class it was meant to absorb. The same applies to the recovery-scan retry in restoreCommit.
Suggestion: Introduce a bounded backoff (e.g. a fixed 1s delay, or capped exponential backoff) between synchronous retry rounds in both commitXidInfos and recoverCheckpointTransactions, so transient RM unavailability is actually absorbed by the retry budget instead of being burned instantly.
**Issue 2**: No integration/real-database test exercising XA driver-level semantics (MAJOR)
Location: XaGroupOpsImplIT.java (connector-jdbc-e2e-part-1); the three new test classes (XaFacadeImplAutoLoadTest, XaGroupOpsImplTest, JdbcSinkAggregatedCommitterTest) are Mockito-based unit tests only
Description: The core of this PR is XA error-code classification (XA_RETRY/XAER_RMFAIL as transient vs XA_RBTRANSIENT as permanent) and the restore/recovery behavior, both of which depend on what the real database driver actually returns (e.g. how MySQL/PostgreSQL surface XAER_NOTA, XA_RETRY, XAER_RMFAIL, and how XAResource.recover() returns driver-specific Xid values). Mock-based unit tests can verify that the classification logic is internally consistent, but they cannot verify that the real drivers behave as assumed. This leaves a gap between the tested behavior and the actual runtime contract.
Suggestion: Re-enable XaGroupOpsImplIT (it remains @Disabled) with a real database, covering at least one end-to-end path where an XA commit/restore failure propagates through to a checkpoint/job failure — this directly validates the failure-propagation behavior this PR is fixing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Propagate JDBC XA commit failures [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on code in PR #11569:
URL: https://github.com/apache/seatunnel/pull/11569#discussion_r3870884724
##########
seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/sink/JdbcSinkAggregatedCommitter.java:
##########
@@ -70,22 +80,76 @@ private void tryOpen() throws IOException {
@Override
public List<JdbcAggregatedCommitInfo> commit(
List<JdbcAggregatedCommitInfo> aggregatedCommitInfos) throws IOException {
+ return commitPreparedTransactions(aggregatedCommitInfos);
+ }
+
+ /**
+ * Reconciles checkpoint XIDs with the resource manager using commit-order evidence. Checkpoint
+ * XIDs from the first still-prepared transaction onward must all be present in the recovery
+ * scan and are replayed strictly. An all-absent batch is treated as already resolved, while an
+ * absent prefix before a still-prepared suffix is treated as already resolved only after that
+ * suffix commits successfully.
+ */
+ @Override
+ public List<JdbcAggregatedCommitInfo> restoreCommit(
+ List<JdbcAggregatedCommitInfo> aggregatedCommitInfos) throws IOException {
+ tryOpen();
+ for (JdbcAggregatedCommitInfo aggregatedCommitInfo : aggregatedCommitInfos) {
+ // Refresh RM evidence for every batch because transactions may be resolved concurrently
+ // during failover while earlier restored batches are being replayed.
+ replayRecoveredCheckpoint(
+ aggregatedCommitInfo.getXidInfoList(), recoverCheckpointTransactions());
+ }
Review Comment:
Thanks for tracing the Zeta lifecycle so carefully — that reasoning is correct, and this is not defending against another Zeta task.
Within a single running `JdbcSinkAggregatedCommitter`, there is no legitimate SeaTunnel-internal actor that could race this scan: restore only replays after the old task group reaches a terminal state, and master failover skips redeploying a `TaskGroupLocation` that is still active, so two live instances of the same committer are never mutating the same XIDs at once. The comment is guarding against something external to Zeta: the resource-manager side of XA resolution. A prepared branch can be resolved independently of the job — a DBA issuing a manual `XA COMMIT`/`XA ROLLBACK` against an in-doubt transaction, or RM-side in-doubt-transaction cleanup tooling that many operators run against long-pending prepared branches (sometimes triggered by exactly this kind of stalled committer retry loop).
That matters concretely here because `aggregatedCommitInfos` passed into `restoreCommit()` is not always a single batch. `SinkAggregatedCommitterTask.restoreState()` flattens every `ActionSubtaskState`'s stored bytes across all not-yet-committed checkpoints (`seatunnel-engine/seatunnel-engine-server/.../SinkAggregatedCommitterTask.java:260-275`), so if several checkpoints completed their prepare phase without their commit notification landing before the failure, restore can walk more than one `JdbcAggregatedCommitInfo` in a single call. `commitXidInfos()` (`JdbcSinkAggregatedCommitter.java:129`) can spend up to `maxCommitAttempts` rounds x 1s backoff per batch, so with N batches the wall-clock gap between "scan taken" and "commit attempted for the last batch" grows with every earlier batch's retries if only one upfront scan is used.
To be precise about what the per-batch refresh actually buys: it narrows that staleness window, it does not close it. There is still an irreducible gap between `recover()` returning and the following `xaFacade.commit()` call for the very same batch — two separate RM round-trips can't be made atomic without RM support this connector doesn't rely on. So this is a defensive freshness measure, not a race-free guarantee, and I'd rather say that plainly than oversell it.
What actually protects correctness if that residual gap is hit: `XAER_NOTA` is not in `TRANSIENT_ERR_CODES` (`XaFacadeImplAutoLoad.java:75-76`), so if a checkpoint XID the fresh scan reported as still-prepared gets resolved externally before the commit call reaches it, `xaFacade.commit()` throws a permanent `JdbcConnectorException`, which surfaces through `result.failed()` -> `throwIfAnyFailed("commit")` (`XaGroupOpsImpl.java:66-72`) and fails the restore loudly rather than silently treating it as resolved. So the residual window can only ever manifest as a rare, operator-visible hard failure, never as silent data loss.
I'll tighten the Javadoc on `restoreCommit()` (currently just says "concurrently") to say it means resolution by an external actor on the resource manager, since as written it does read like it could mean another Zeta task, which was a completely fair thing to question.
##########
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-part-1/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/xa/XaGroupOpsImplIT.java:
##########
Review Comment:
That's a reasonable ask, and I don't want to wave it off — but I'd treat it as a valuable follow-up rather than a blocker for this PR.
What's already exercising the real failure/restore path today: `XaGroupOpsImplIT` in this same file runs against a real MySQL testcontainer (not a mock), and `testCommitFailurePropagatesThroughAggregatedCommitter` (line 126) drives an actual XA prepare -> forced commit failure -> propagation through `JdbcSinkAggregatedCommitter`. On top of that, `JdbcSinkAggregatedCommitterTest` has unit coverage for every reconciliation branch this PR introduces: recovered-by-value matching, unrelated recovered XIDs, already-resolved prefixes, no-evidence missing gaps failing closed, and bounded transient-retry behavior for both commit and the recovery scan itself.
What a full "partial XA commit -> kill Zeta -> restart -> verify recovery" E2E would add on top of that is genuine end-to-end confidence that the wiring between `SinkAggregatedCommitterTask.restoreState()` and this committer is correct under an actual cluster restart, which the IT/UT layer can't fully substitute for. The reason I didn't reach for it in this PR is that this exact class of test (kill a running Zeta cluster mid-checkpoint, actually restart it, and assert final DB state) is exactly the shape of E2E the project has been trying to get away from — controlling when the kill lands relative to the XA prepare/commit boundary without a `Thread.sleep`-based timing guess is nontrivial, and a flaky version of this test would be worse for the project than not having it at all.
I'm not opposed to adding it as a separate, deliberately-scoped follow-up PR with proper deterministic synchronization (for example gating the kill on an observable committer-side signal instead of a timing guess) — I'd rather get that design right in its own review than fold it into this one under time pressure. Happy to file the follow-up issue if that's useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Propagate JDBC XA commit failures [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11569:
URL: https://github.com/apache/seatunnel/pull/11569#issuecomment-5466087061
Note: this PR is authored by me (DanielLeens), so GitHub blocks a self-review submission. Posting this as a plain PR comment instead of a formal review, per project convention for self-authored PRs (same as my previous rounds on this thread). Since `git diff` between my last comment and the current head is not empty this time, this is a genuine incremental round, not a repeat.
New activity since my last comment (`a9257e75bf1a`, then Issue 1 posted against it): three new commits from @davidzollo — `5ec277d8be97` ("[Docs][Connector-V2] Clarify XA recovery cleanup risk"), `c5f4ec0a0d90` ("[Docs][Connector-V2] Clarify XA recovery concurrency"), and `d2bc272f3641` ("[Chore][Connector-V2] Refresh CI", current head) — plus a new reply from @davidzollo on @dybyte's Q1 thread. I re-verified each of these directly against source rather than taking the commit messages or the reply at face value.
# What Problem Does This PR Solve?
Unchanged from prior rounds: before this patch, a permanent JDBC XA commit failure could be silently swallowed (`throwIfAnyFailed("commit")` was disabled, and `wrapException()` threw `TransientXaException` pre-wrapped inside `JdbcConnectorException`, making the retryable-error branch unreachable). This PR restores failure propagation, fixes retryable/permanent XA error classification, bounds retries within one invocation, and reconciles restored checkpoint XIDs against a live `xaFacade.recover()` scan instead of inferring success from an absent XID alone.
# 1. Code Change Review
## 1.1 Core Logic Analysis
I pulled the three new commits as raw diffs from the GitHub API and cross-checked them against the current head's full source for `JdbcSinkAggregatedCommitter.java`, `XaGroupOpsImpl.java`, `XaFacadeImplAutoLoad.java`, and `GroupXaOperationResult.java` rather than trusting the commit subjects:
- **`5ec277d8be97`** is exactly the one-line migration-guide addition I recommended as the "Best improvement" for my own Issue 1 (absence-from-recovery-scan cannot distinguish "committed by us" from "rolled back externally"; only the gap/fail-closed case was previously disclosed). Both `docs/en` and `docs/zh` `incompatible-changes.md` now state plainly that XA recovery cannot tell a SeaTunnel-committed XID from one rolled back or removed by an external actor, and that operators should not run external cleanup against SeaTunnel-owned prepared branches while a job may still be restored, without coordinating with recovery. This closes Issue 1 as I scoped it — a documentation fix, not a code fix, since the underlying gap is information-theoretic to XA itself, not an implementation defect.
- **`c5f4ec0a0d90`** tightens the `restoreCommit()` Javadoc to name the external actor explicitly ("a DBA or RM cleanup process"), state plainly that this is *not* a second Zeta committer, and say outright that the per-batch recovery-scan refresh "narrows, but cannot eliminate, the gap before the following XA commit call." I checked this against @davidzollo's reply on @dybyte's Q1 thread ("Addressed in c5f4ec0a0d") and it is accurate — the new Javadoc text matches exactly what his inline reply and my own earlier reply in that same thread already committed to putting in writing. This is a comment-only change; no executable logic moved.
- **`d2bc272f3641`** (current head) — verified via `gh api repos/apache/seatunnel/commits/<sha>` that this commit has zero changed files. It is a genuine empty CI-retrigger commit, not a disguised code change.
I independently re-verified the load-bearing invariants against the current head's actual source rather than re-trusting my own prior rounds:
- `GroupXaOperationResult.hasNoFailures()` returns `false` if *either* `failure` (permanent) or `transientFailure` (transient) is present; `throwIfAnyFailed` only throws on the permanent `failure` field. Combined with `XaGroupOpsImpl.commit()`'s loop guard `i.hasNext() && (result.hasNoFailures() || allowOutOfOrderCommits)` (both production call sites pass `allowOutOfOrderCommits=false`), the loop stops at the first failure of any kind and every un-iterated XID is appended to `forRetry` via the same iterator, preserving order. This is the exact ordering invariant `JdbcSinkAggregatedCommitter.findFirstRecoveredIndex`/`replayRecoveredCheckpoint` (lines 207-239) depends on, and it holds in the source I read directly.
- `XaFacadeImplAutoLoad.wrapException` returns (does not throw) a bare `TransientXaException` for `TRANSIENT_ERR_CODES = {XA_RETRY, XAER_RMFAIL}` and a `JdbcConnectorException` for everything else, including `XAER_NOTA` and `XA_RBTRANSIENT`; every one of `execute()`'s two throw sites and both `Command.fromRunnable*` closures does `throw wrapException(...)`, so `XaGroupOpsImpl.commit()`'s `catch (XaFacade.TransientXaException e)` is genuinely reachable. This confirms the core claim of the PR (the pre-fix bug: the exception used to arrive pre-wrapped inside `JdbcConnectorException`, so that catch clause was dead code).
Both of these match the six prior rounds' conclusions; I re-derived them from the current head's source rather than assuming they still hold, since re-verifying invariants after every commit — even a docs-only one — is the point of doing another pass at all.
## 1.2 Compatibility Impact
Unchanged: partially incompatible, and now more completely disclosed. `5ec277d8be97` extends the existing migration-guide breaking-change entry (both `en` and `zh`) to also cover the prefix/all-absent inference gap, closing the one disclosure hole this thread had identified. No checkpoint state schema changed; no source-incompatible API changes in this round.
## 1.3 Performance / Side-Effect Analysis
No change this round — both new substantive commits are documentation/Javadoc only, and the third is an empty commit. Nothing new to assess beyond what was already established in prior rounds (bounded restart-only-path `recover()` scan cost, 1s backoff between retry rounds, O(N) `containsEquivalentXid` scan per batch).
## 1.4 Error Handling and Logging
No new issues. The two threads open against the current head are both now answered in the code/docs, not just in comment text:
- @dybyte's Q1 (concurrent-resolution actor) — answered in the thread reply and now also captured durably in the `restoreCommit()` Javadoc via `c5f4ec0a0d90`, so the explanation survives independent of the PR discussion.
- My own Issue 1 (prefix/all-absent disclosure gap) — closed via the migration-guide addition in `5ec277d8be97`.
@dybyte's Q2 (non-blocking ask for a kill-and-restart E2E covering the full checkpoint -> failure -> restore path) remains an open, explicitly-scoped follow-up rather than something this round changed — I stand by the reasoning in my prior reply that a timing-guess-free version of that E2E deserves its own review rather than being folded in here under time pressure.
# 2. Code Quality Assessment
## 2.1 Coding Standards
Both new substantive commits are comment/doc-only and read clearly; no new core method or field was added this round, so no new comment-coverage gap to flag.
## 2.2 Test Coverage and Test Stability: Stable
No test files changed in this round. Unchanged from my prior assessment: `XaFacadeImplAutoLoadTest`/`XaGroupOpsImplTest` exercise real `XAException` error codes against mocks, `JdbcSinkAggregatedCommitterTest` covers every reconciliation branch (prefix-skip, all-absent skip, fail-closed-on-gap, transient retry, retry-exhaustion), and `XaGroupOpsImplIT` now runs a real MySQL Testcontainer end to end instead of being `@Disabled`.
## 2.3 Documentation Updates
Improved this round, not just unchanged: the migration guide gap I flagged as Issue 1 is now closed in both `docs/en` and `docs/zh`. `docs/en/connectors/sink/Jdbc.md`/`docs/zh` remain accurate against the code as previously verified.
# 3. Architectural Soundness
## 3.1 Elegance of the Solution: Precise fix
Unchanged.
## 3.2 Maintainability
Unchanged.
## 3.3 Extensibility
Unchanged.
## 3.4 Historical-Version Compatibility
Unchanged: no wire/state-format change; the operational risk is disclosed, and this round makes that disclosure more complete rather than introducing anything new to assess.
# 4. Issue Summary
| Number | Issue | Location | Severity |
| --- | --- | --- | --- |
| 1 (closed) | Absent-from-recovery-scan XIDs could not be distinguished from an external rollback, and only the gap/fail-closed case was disclosed | `docs/en(zh)/introduction/concepts/incompatible-changes.md` | Was Medium — resolved by `5ec277d8be97` |
| N/A-2 | `XaFacade.commit(xid, ignoreUnknown=true)` still has no production caller after evidence-based replay replaced `XAER_NOTA`-as-success inference (li3zhi4, still open) | `JdbcSinkAggregatedCommitter.java`, `XaFacadeImplAutoLoad.java` | Low |
| N/A-3 | PR description still overstates the all-absent case as "fails closed" when code/docs treat it as already-resolved (li3zhi4, still open, description-only) | PR description text | Low |
No new blocking issues found on this head.
# 5. Merge Recommendation
### Conclusion: Ready to merge
From a source-correctness standpoint, this round's two substantive commits do exactly what their messages and the accompanying thread replies claim — I verified both against the actual diffs and current source rather than trusting the commit subjects, and the third commit is a genuinely empty CI retrigger. My own Issue 1 is now closed by documentation rather than left as a recommendation. No new code-level issue surfaced.
Restating the process status precisely, since it's easy to overstate: `reviewDecision` is currently `CHANGES_REQUESTED`, driven solely by @nzw921rx's 2026-07-27 review — which, as established in my 2026-08-29 full re-review, predates the recovery-scan reconciliation mechanism entirely and targets a design that no longer exists in this form. @davidzollo's 2026-08-06 `CHANGES_REQUESTED` no longer counts toward `reviewDecision` because his latest review on this thread is `COMMENTED` (his concrete non-idempotency scenario was fixed in code back in `97b0942d7b2a`, and he has continued actively pushing fixes and replying since) — but I want to be precise that GitHub superseding his review state for computation purposes is not the same as him having formally said "resolved, no objection," so I'm not claiming more than the mechanical fact here. @dybyte's `APPROVED` review was auto-dismissed by branch protection when the `5ec277d8be97` commit landed, which is expected push-triggered be
havior, not a new objection — nothing in that thread asked for anything this round didn't already close out.
CI: the `Build` check on the current head (`d2bc272f3641`) is still in progress on the fork as I write this (run `33278167729`, ~3.5h in). Three jobs have already failed early — `updated-modules-integration-test-part-2`, `updated-modules-integration-test-part-3`, and `doris-connector-it` (all on JDK 8, ubuntu-latest) — but their logs aren't retrievable yet while the overall run is still active, so I can't pull the exact failure lines this time the way I did in my previous four CI-triage rounds on this same PR. What I can say factually: the first two of those three job buckets have failed and been diagnosed as unrelated Maven Central network flakes or an unrelated `PostgresCDCIT` timing flake in every prior round on this exact PR (this PR's diff has never touched `connector-jdbc-e2e-part-2`, `connector-milvus`, `connector-cdc-postgres`, or `connector-doris`), so this is consistent with — but not yet independently re-confirmed as — the same recurring infra-flake pattern. I'm
flagging that distinction rather than asserting the current failures are flakes before I've actually read their logs.
1. Blockers (process, not code): @nzw921rx's stale `CHANGES_REQUESTED` needs to be revisited against the current head and either re-affirmed with a concrete remaining objection or dismissed/updated by them or a maintainer with write access — I cannot do this myself as the PR author. CI is still running on the current head with three early failures whose logs aren't yet available; once it completes, those logs should be pulled to confirm they match the established unrelated-infra pattern before assuming a rerun is all that's needed.
2. Recommended fixes (non-blocking): the two low-severity carryovers from li3zhi4 (PR description wording, unused `ignoreUnknown=true` path) remain open and are cheap to fold in whenever this PR is touched again.
No alternative implementation approach changes here; the commit-order-evidence design remains the right shape given that Zeta doesn't persist partial-commit state across a checkpoint restart today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Propagate JDBC XA commit failures [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11569:
URL: https://github.com/apache/seatunnel/pull/11569#issuecomment-5339594060
Current head `03f7fd4347f7` has addressed the previous XA restore blocker and is ready for human re-review from my side.
What changed after the 2026-08-06 review:
- `restoreCommit()` no longer treats every `XAER_NOTA` as idempotent success.
- Restore now runs an XA `recover()` scan, normalizes driver-specific `Xid` values by `formatId` + GTRID + BQUAL, and commits only checkpoint-owned XIDs that are still present in the resource manager recovery scan.
- A missing checkpoint XID is only skipped when it is an already-resolved prefix before the first still-prepared transaction in the same commit batch; missing gaps/tails fail closed instead of being reported as successful.
- Added regression coverage for recovered-by-value matching, unrelated recovered XIDs, already-resolved prefixes, no-evidence missing XIDs, missing gaps after a recovered transaction, transient recovery-scan retry, and bounded commit retry behavior.
- Docs now describe the JDBC XA restore/recovery behavior and incompatible behavior change.
Current CI note: the latest required Build on this head failed in `paimon-connector-it`, specifically `PaimonSinkCDCIT.testSinkWithIncompatibleSchema`, with a Paimon schema mismatch / container thread timeout. That job is outside this PR's JDBC XA source/test scope, so I do not see it as a code blocker for this PR.
Could reviewers please re-check the latest head and clear the stale review gate if the restore reconciliation looks good?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Propagate JDBC XA commit failures [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11569:
URL: https://github.com/apache/seatunnel/pull/11569#issuecomment-5462160229
Note: this PR is authored by me (DanielLeens), so GitHub blocks a self-review submission. Posting this as a plain issue comment instead of a formal PR review, per project convention for self-authored PRs (same as my previous rounds on this thread).
I re-checked activity since my last comment (`a9257e75bf1a`, ~10 hours ago): `git diff` between that head and the current head is empty — no new commits, no new reviews, no new discussion. Given that, I'm not going to re-post the ~2000-word from-scratch trace verbatim again; it would add nothing over what's already sitting immediately above this comment on the same code. What follows is condensed to (a) an independent spot-check of my own prior conclusions using a different method than before, and (b) one genuinely new finding that none of the six review rounds on this thread (mine included) named explicitly.
# What Problem Does This PR Solve?
Unchanged from prior rounds: before this patch, a permanent JDBC XA commit failure could be silently swallowed (`throwIfAnyFailed("commit")` was disabled, and `wrapException()` always threw `TransientXaException` pre-wrapped inside `JdbcConnectorException`, making the retryable-error branch unreachable). This PR restores failure propagation, fixes retryable/permanent XA error classification, bounds retries within one invocation, and reconciles restored checkpoint XIDs against a live `xaFacade.recover()` scan instead of inferring success from an absent XID alone.
# 1. Code Change Review
## 1.1 Core Logic Analysis
**Independent verification of the failure-propagation fix** (`XaFacadeImplAutoLoad.java`, `XaGroupOpsImpl.java`): confirmed by reading every `wrapException` call site, not just the two I discussed before. All four sites (`execute()`'s `orElseThrow`, `execute()`'s catch-all `throw`, and both `Command.fromRunnable`/`fromRunnableRecoverByWarn` closures) consistently do `throw wrapException(...)`. Since `wrapException` now *returns* a `RuntimeException` (either `TransientXaException` or `JdbcConnectorException`) instead of throwing it pre-wrapped, every one of those four call sites now throws the correct concrete type, so `XaGroupOpsImpl.commit()`'s `catch (XaFacade.TransientXaException e)` (previously dead code, since the exception used to arrive as a `JdbcConnectorException` wrapping a `TransientXaException` inside its cause, not as a `TransientXaException` itself) is genuinely reachable. `TRANSIENT_ERR_CODES` correctly moved from `{XA_RBTRANSIENT}` to `{XA_RETRY, XAER_RMFAIL}` —
`XA_RB*` codes mean the branch is already rolled back (permanent, nothing to retry), `XA_RETRY` means the call had no effect and is safe to reissue. This is real XA semantics, not a cosmetic rename.
**Independent verification of the ordering invariant** the restore reconciliation depends on: `XaGroupOpsImpl.commit()`'s loop condition is `i.hasNext() && (result.hasNoFailures() || allowOutOfOrderCommits)`, both production call sites pass `allowOutOfOrderCommits=false`, and `hasNoFailures()` flips to `false` on *either* a permanent or a transient failure. So within one invocation the loop stops at the first failure of any kind, and every subsequent un-iterated XID is appended to `forRetry` untouched (`result.getForRetry().addAll(xids)` after the loop, using the same `Iterator`, so relative order is preserved). Across retry rounds in `commitXidInfos`'s `while (!pending.isEmpty())` loop, `pending` is rebuilt from `result.getForRetry()` each round, so order is preserved across rounds too. This confirms the property `replayRecoveredCheckpoint`/`findFirstRecoveredIndex` (`JdbcSinkAggregatedCommitter.java:205-237`) relies on: a later XID in a checkpoint's list is only ever committed a
fter every earlier XID in that same list already committed. I traced this end-to-end myself rather than trusting the Javadoc's claim, and it holds on the current head.
**New finding — Issue 1 below.** Tracing the two "absorb an absence" branches (all-absent batch, and the skipped prefix before a still-prepared suffix) surfaced a gap that none of the six review rounds on this PR (nzw921rx, davidzollo, li3zhi4, JeremyXin, dybyte, or my own five prior rounds) named explicitly, though dybyte's Q1 thread on `JdbcSinkAggregatedCommitter.java:103` came close from a different angle (concurrent resolution timing, which I answered fully for the still-prepared/suffix case in that thread — external resolution there surfaces as a loud `XAER_NOTA`-is-permanent failure, since `XAER_NOTA` isn't in `TRANSIENT_ERR_CODES`). What I hadn't examined is the *prefix/all-absent* case specifically:
### Issue 1: Absence from the recovery scan cannot distinguish "committed by us" from "rolled back by someone else," and only the fail-closed (gap) case is disclosed to operators
- **Location**: `JdbcSinkAggregatedCommitter.java:205-237` (`replayRecoveredCheckpoint`), and the corresponding entries in `docs/en(zh)/introduction/concepts/incompatible-changes.md` / `docs/en(zh)/connectors/sink/Jdbc.md`
- **Problem**: Per the XA spec, a resource manager forgets a branch identically whether it was previously *committed* or *rolled back* — both outcomes make the branch vanish from `recover()`, and both return `XAER_NOTA` if a client tries to act on it again. The prefix-skip and all-absent-batch branches use "absent from the scan" as their *only* signal, and both optimistically assume "already committed by us in a prior aborted attempt." Under this PR's own architecture that assumption is well-justified for the *normal* failure mode it fixes (a batch's own earlier commit attempt got interrupted by an unrelated restart) — but as dybyte's Q1 thread already established, the resource manager can also be resolved by an actor external to Zeta: a DBA or an automated stuck-prepared-transaction cleanup job. If that external actor issues `XA ROLLBACK` instead of `XA COMMIT` against a branch that actually still belongs to a live (merely delayed) SeaTunnel job — a realistic scenario after
a long outage, since long-pending prepared XA transactions are a well-known operational headache that provokes exactly this kind of manual/automated cleanup — this code has no way to tell the difference, and unlike the suffix/gap case, the prefix/all-absent branches never re-attempt a commit on the skipped XIDs, so there is no `XAER_NOTA`-permanent-failure signal to surface the mistake. The checkpoint is reported as fully resolved while that XID's writes are permanently gone. The `log.warn` at `JdbcSinkAggregatedCommitter.java:212-214` and `:232-235` is the only operational trace, and it reads identically whether the skip was correct (the very common case) or masking a real loss (the rare case) — an operator cannot tell which happened from the log alone.
- **Why this isn't a reason to reject the design**: the alternative — failing closed on every absence, including the all-absent case — is exactly the infinite-restart-deadlock bug davidzollo's 2026-08-06 review found in an earlier revision of this PR (a job stuck forever re-attempting a batch that was legitimately already committed). XA fundamentally cannot distinguish "resolved by commit" from "resolved by rollback" from the RM's response alone without extra durable state that Zeta doesn't checkpoint today (the PR description is upfront about this constraint), so some assumption is unavoidable, and "assume committed" is the correct default given SeaTunnel's own aborts never touch a checkpoint-committed XID's namespace (each checkpoint/subtask generates distinct XIDs, so self-inflicted confusion is ruled out — I checked this specifically since it would have been a much worse bug).
- **Potential risk**: silent, permanent, undetectable loss of one XID's writes in the narrow window where an external actor rolls back (rather than commits) a prepared transaction this job still expects to restore — with no distinguishing signal from a routine, correct skip.
- **Best improvement**: this doesn't need a code change (there's no code fix for an information-theoretic gap), but the incompatible-changes migration guide currently only tells operators to investigate a *gap after a recovered XID* before upgrading — it should say the same for the all-absent/prefix-skip case: that "already resolved" there is inferred, not verified, and that operators who run external stuck-transaction cleanup against a resource manager also used by SeaTunnel should exclude branches whose global-transaction-id prefix matches SeaTunnel's `XidGenerator` format, or otherwise coordinate with job restart timing. A one-line addition to the existing migration guide entry closes this cheaply.
- **Severity**: Medium (real residual risk, correctly scoped as inherent-to-XA rather than a implementation bug; recommended before merge, not a hard blocker given the log-level signal that does exist and the fact this PR is a strict improvement over the pre-existing status quo either way)
- **Raised by another reviewer**: No — new angle on a concern dybyte's Q1 partially opened; my own reply in that thread fully closed the suffix/still-prepared case but did not address the prefix/all-absent case, which is the part I'm completing here.
## 1.2 Compatibility Impact
Unchanged from prior rounds: Partially incompatible, and correctly disclosed for the gap/fail-closed scenario (see Issue 1 for the one additional disclosure gap). No checkpoint state schema changed (`XidInfo` serialization untouched); `TransientXaException`'s constructor widened from package-private to `public`; `XaGroupOps.commit()` gained Javadoc only. No source-incompatible API changes.
## 1.3 Performance / Side-Effect Analysis
Unchanged: `restoreCommit()`'s `recover()` scan is a bounded, restart-only-path cost with its own retry budget; the 1s `Thread.sleep` backoff between synchronous retry rounds (added in response to JeremyXin's 2026-08-21 review) only fires under genuine XA/RM instability; `containsEquivalentXid` is an O(N) scan per checkpoint XID against a `Set<XidKey>`, bounded by batch size.
## 1.4 Error Handling and Logging
No new issues beyond Issue 1 above. I independently re-verified `throwIfAnyReachedMaxAttempts` in `XaGroupOpsImpl.java` and `commitXidInfos`'s outer bounded loop in `JdbcSinkAggregatedCommitter.java` both actually terminate (covered by `testCommitFailsWhenRetryRoundsNeverDrain`) rather than looping forever on a permanently-stuck resource manager.
**CI diagnosis for the current head (`a9257e75bf1a`)** — unchanged from my comment immediately above, and I independently re-verified it rather than taking my own prior word for it: pulled the actual `unit-test (8, ubuntu-latest)` job log from the fork run (`DanielLeens/seatunnel` run `33238169880`, job `99063544383`) myself. Confirmed the failure is exactly `[ERROR] CoordinatorServiceTest.testClearCoordinatorServiceDropsPendingJobsUnderRejectStrategy:642 » ConditionTimeout`, in `seatunnel-engine-server`, a module this PR's diff never touches (this PR's diff is scoped to `connector-jdbc`'s XA classes/tests, `connector-jdbc-e2e-part-1`, and docs — confirmed via `git diff --stat` against the correct merge-base `43fe63b1fca`). The other three failing jobs (`updated-modules-integration-test-part-3`/`part-2` on JDK 8 and 11) were already diagnosed as Maven Central dependency-resolution resets and an unrelated `PostgresCDCIT` Awaitility timeout, in modules this PR does not touch ei
ther. This is infra flake, not a regression from this PR — needs a CI rerun, not a code change.
# 2. Code Quality Assessment
## 2.1 Coding Standards
Unchanged: all new/changed core methods carry Javadoc explaining intent and the fail-closed/already-resolved distinction. No gaps found.
## 2.2 Test Coverage and Test Stability: Stable
Unchanged rating, independently re-confirmed by reading the full method list in `JdbcSinkAggregatedCommitterTest.java`: dedicated coverage exists for recovered-by-value matching, per-batch recovery-scan refresh, prefix-skip-after-recovered-suffix, all-absent-batch skip, fail-closed-on-gap, transient recovery-scan retry, bounded commit-retry completion, and retry-exhaustion. `XaGroupOpsImplIT` exercises a real MySQL testcontainer end to end (addresses JeremyXin's Issue 2). No test exists for the Issue 1 scenario above, which is expected — there is no deterministic way to unit-test "an external actor rolled back instead of committed," since the code itself cannot observe the difference; this is a documentation gap, not a missing-test gap.
## 2.3 Documentation Updates
Current docs accurately describe the mechanism (verified by diff, not just PR description) but have the one gap named in Issue 1.
# 3. Architectural Soundness
## 3.1 Elegance of the Solution: Precise fix
Unchanged: scoped tightly to the XA commit/restore path, no leakage into `seatunnel-api` or the engine.
## 3.2 Maintainability
Unchanged: small, single-purpose, documented helpers (`XidKey`, `backoffBeforeRetry`, the commit/restoreCommit split).
## 3.3 Extensibility
Unchanged: reconciliation operates on canonical XID values via the existing `XaFacade`/`XaGroupOps` abstractions.
## 3.4 Historical-Version Compatibility
Unchanged: no wire/state-format change; the risk is purely operational and disclosed for the gap case (Issue 1 extends this disclosure to the prefix/all-absent case).
# 4. Issue Summary
| Number | Issue | Location | Severity |
| --- | --- | --- | --- |
| 1 | Absent-from-recovery-scan XIDs (prefix skip, all-absent batch) are assumed committed with no way to distinguish an externally-issued rollback; only the gap/fail-closed case is disclosed in the migration guide | `JdbcSinkAggregatedCommitter.java:205-237`, `docs/en(zh)/introduction/concepts/incompatible-changes.md` | Medium |
| N/A-2 | `XaFacade.commit(xid, ignoreUnknown=true)` has no production caller after evidence-based replay replaced `XAER_NOTA`-as-success inference (li3zhi4, still open) | `JdbcSinkAggregatedCommitter.java`, `XaFacadeImplAutoLoad.java` | Low |
| N/A-3 | PR description overstates the all-absent case as "fails closed" when code/docs treat it as already-resolved (li3zhi4, still open, description-only) | PR description text | Low |
# 5. Merge Recommendation
### Conclusion: Ready to merge
Issue 1 is a real, previously-unarticulated residual risk, but it's an inherent consequence of XA's own commit/rollback ambiguity rather than an implementation defect, it's already partially mitigated by WARN-level logging, and the fix is a one-line addition to an already-thorough migration guide rather than a code change — I don't think it should block merge, but I'd like it addressed before this lands given how much this PR already invests in disclosing the gap-case risk to operators.
Restating what hasn't changed: both outstanding `CHANGES_REQUESTED` reviews (nzw921rx 2026-07-27, davidzollo 2026-08-06) target concerns the current implementation structurally resolves — I re-verified this myself from source rather than assuming my prior rounds still hold. That's a process blocker, not a code blocker: I'm the PR author and cannot dismiss another reviewer's review or approve my own PR; this needs nzw921rx/davidzollo to re-review the current head, or a maintainer with write access to dismiss the stale reviews. @dybyte's `APPROVED` (2026-08-28, same head modulo the empty CI-retry commit) is independent confirmation the current implementation is sound.
1. Blockers (process, not code): the two stale `CHANGES_REQUESTED` reviews need to be revisited/dismissed by their authors or a maintainer. CI needs a clean rerun given the four confirmed-unrelated failure causes above (independently re-verified this round, not just carried forward).
2. Recommended fixes (non-blocking): add the one-line migration-guide disclosure for Issue 1; the two low-severity carryovers from li3zhi4 (PR description wording, unused `ignoreUnknown=true` path).
No alternative implementation approach changes here; the commit-order-evidence design remains the right shape given that Zeta doesn't persist partial-commit state across a checkpoint restart today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org