You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "davidzollo (via GitHub)" <gi...@apache.org> on 2026/08/18 04:05:42 UTC
[I] [Bug][Connector-V2][Postgres-CDC] Committed-offset restore can silently skip a row committed after savepoint (slot LSN advances past it) [seatunnel]
davidzollo opened a new issue, #11847:
URL: https://github.com/apache/seatunnel/issues/11847
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues) and found no similar issues.
### What happened
While stabilizing PR #11556 we hit a reproducible silent data-loss pattern in `PostgresCDCIT.testPostgresCdcSnapshotOnlyAndCommittedOffsetStartupModes` (committed-offset startup mode, no exactly-once): after a savepoint + restore, a row committed between the savepoint's offset commit and the stop is never delivered to the sink, and the replication slot's committed LSN advances **past** that row's commit LSN, so the loss is permanent (a later restart can no longer replay it).
Evidence timeline from the failing run (fork run `31962112336` on the PR branch, both JDK 8 and 11 failed identically; log windows archived):
- savepoint checkpoint 4 commits slot offset `0/2283BE0` (36191200) at 18:49:20.8; `SAVEPOINT_DONE` 18:49:21.335
- row `id=15` inserted ~18:49:21.5 (after savepoint completion, before restore)
- restore at 18:49:24.6 with startupOffset `{lsn=lsn_commit=lsn_proc=36191200, txId=810}`
- Debezium restarts (`START_REPLICATION ... 0/2283BE0` twice = WalPositionLocator search + replay phases); checkpoints 5-27 all complete
- the reader's committed offset advances exactly once, `0/2283BE0 -> 0/2283D18` (36191512, past id=15's commit), then freezes
- restored job's "Read Count So Far" stays at the pre-savepoint total (93) forever: the restored source emitted zero rows; the sink poll for id=15 times out after 120s
### Scope observed so far (important caveats)
- The failure appeared on the PR branch immediately after merging dev range `163a36ac..da7b1e01` (merge commit `86bcff279f6`): the identical scenario had passed 3/3 runs on the same PR before that merge (e.g. runs `31235087244`, `31290962731`, where the restored job delivered the post-savepoint row), and failed 2/2 after.
- Static analysis of the PR's own deltas found none reachable in this job's path (all gated on exactly-once and/or INITIAL startup, which this test does not use; the test method itself is byte-identical to dev's version from #11380).
- However, dev's own push CI has been passing the shard containing `PostgresCDCIT`, so this may be a timing-sensitive race that the PR branch amplifies rather than a deterministic dev regression. The dev-side changes in that range touching this path include the cdc-base incremental fetcher/reader rework (#11618) and Zeta restore-from-latest-checkpoint (#11421).
### Related latent defect (aggravating)
`PostgresWalFetchTask.execute` never checks `errorHandler.getProducerThrowable()` after `PostgresStreamingChangeEventSource.execute` returns, and Debezium 1.9.8 swallows streaming failures into the error handler — a wedged/dead replay after restore is therefore invisible to the job. (The newer backfill read task in #11556 does check it.) Two hardening candidates independent of the root cause:
1. surface `errorHandler.getProducerThrowable()` in the incremental WAL fetch path;
2. avoid committing/flushing a slot LSN past the last row-emitting record's offset while a restored stream has emitted nothing — that converts the silent skip from permanent loss into a replayable gap.
### What you expected to happen
A row committed after the savepoint's offset commit must be delivered by the restored job (or at minimum remain replayable in the slot); the committed slot LSN must not advance past an undelivered row.
### How to reproduce
Run `PostgresCDCIT.testPostgresCdcSnapshotOnlyAndCommittedOffsetStartupModes` on a branch containing dev `da7b1e01` (observed on PR #11556's branch; 2/2 failures there). The savepoint -> insert -> restore choreography in that test is the reproduction.
### SeaTunnel Version
dev (observed at `da7b1e01`-era merge; PR branch head `506e295e3037`)
### Zeta or Flink or Spark Version
Zeta
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][Connector-V2][Postgres-CDC] Committed-offset restore can silently skip a row committed after savepoint (slot LSN advances past it) [seatunnel]
Posted by "zhang-arvin (via GitHub)" <gi...@apache.org>.
zhang-arvin commented on issue #11847:
URL: https://github.com/apache/seatunnel/issues/11847#issuecomment-5337472939
I would like to work on this issue. Will investigate and submit a PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][Connector-V2][Postgres-CDC] Committed-offset restore can silently skip a row committed after savepoint (slot LSN advances past it) [seatunnel]
Posted by "zhang-arvin (via GitHub)" <gi...@apache.org>.
zhang-arvin commented on issue #11847:
URL: https://github.com/apache/seatunnel/issues/11847#issuecomment-5335202772
I would like to work on this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][Connector-V2][Postgres-CDC] Committed-offset restore can silently skip a row committed after savepoint (slot LSN advances past it) [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on issue #11847:
URL: https://github.com/apache/seatunnel/issues/11847#issuecomment-5343014193
Additional evidence that this is a dev-level race, not something specific to PR #11556:
The same test (`PostgresCDCIT.testPostgresCdcSnapshotOnlyAndCommittedOffsetStartupModes`, dev's unmodified copy, ConditionTimeout at the post-restore row assertion, elapsed ~182s) just failed on **PR #11382** ("Enforce schema change drain before sink DDL" — an engine-side PR that does not touch the Postgres CDC connector at all) in fork run 32094650264, job `all-connectors-it-6 (8, ubuntu-latest)`, on a branch whose merge base is current dev (2026-08-18).
That makes two unrelated branches reproducing the identical post-restore zero-emission symptom since the `163a36ac..da7b1e01` window landed, alongside dev's own push CI passing the test intermittently — consistent with a timing race in the restart/offset-commit interplay rather than a deterministic break. Raising visibility accordingly: on a loss-prone race, an intermittently green CI is the dangerous outcome, since the failure mode is silent row loss in committed-offset restores.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][Connector-V2][Postgres-CDC] Committed-offset restore can silently skip a row committed after savepoint (slot LSN advances past it) [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on issue #11847:
URL: https://github.com/apache/seatunnel/issues/11847#issuecomment-5358316360
Thanks @zhang-arvin for volunteering — assigned to you, please go ahead.
To recap the scope so we stay focused: the problem is that a committed-offset restore can permanently skip a row when it commits between the savepoint's offset commit and the stop, because the replication slot's confirmed LSN advances past that row's commit LSN without the row ever being emitted.
@davidzollo's latest data point reinforces that this is a dev-level timing race rather than something introduced by a single PR: the same `PostgresCDCIT.testPostgresCdcSnapshotOnlyAndCommittedOffsetStartupModes` failure reproduced on an unrelated engine-side branch (fork run 32094650264, job `all-connectors-it-6 (8, ubuntu-latest)`), with the symptom appearing since the `163a36ac..da7b1e01` window while dev's own CI passes intermittently. As he notes, intermittent green is the worst case here — the failure mode is silent row loss.
Concrete asks for @zhang-arvin:
1. Try to reproduce the race locally, ideally with a deterministic hook or injected delay between the offset commit during savepoint and the actual stop, so we can confirm the confirmed-LSN-advances-past-uncommitted-emission sequence.
2. Once confirmed, share your analysis of where the fix belongs (deferring the slot's confirmed LSN advance vs. tightening the commit/stop ordering) before the PR, so we can align on the approach.
3. Please add or extend a test that fails reliably under the injected timing, not just the existing intermittent IT — otherwise we can't distinguish a real fix from a lucky green run.
Please post your reproduction findings here when you have them. Thanks both!
<!-- streview-comment:394 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][Connector-V2][Postgres-CDC] Committed-offset restore can silently skip a row committed after savepoint (slot LSN advances past it) [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo closed issue #11847: [Bug][Connector-V2][Postgres-CDC] Committed-offset restore can silently skip a row committed after savepoint (slot LSN advances past it)
URL: https://github.com/apache/seatunnel/issues/11847
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][Connector-V2][Postgres-CDC] Committed-offset restore can silently skip a row committed after savepoint (slot LSN advances past it) [seatunnel]
Posted by "zhang-arvin (via GitHub)" <gi...@apache.org>.
zhang-arvin commented on issue #11847:
URL: https://github.com/apache/seatunnel/issues/11847#issuecomment-5371865951
@SEZ9 Thanks for the clear scope. I have completed the analysis and submitted a fix in PR #11864. Here is a summary of the findings and approach for each of your three asks:
### 1. Reproduction analysis
I did not reproduce the race locally with a timing hook — instead I used static code analysis to trace the full savepoint→restore path, and found **three deterministic bugs** (none require a timing hook to trigger — they are logic errors that make the race *inevitable* rather than accidental):
**Bug 1 — `commitCurrentOffset` sends the wrong LSN to Debezium**
`PostgresWalFetchTask.commitCurrentOffset()` calls `offset.getLsn()` which reads from `SourceInfo.LSN_KEY` (`"lsn"`), but Debezium's `commitOffset` expects the `LAST_COMMIT_LSN_KEY` (`"lsn_commit"`) field. This means the slot's confirmed LSN is advanced to the mid-transaction LSN of the last processed event, not its commit boundary — so on a subsequent restart, the slot has already moved past rows that committed between the savepoint and the stop.
**Bug 2 — `shouldEmit` uses strict `isAfter` instead of `isAtOrAfter`**
`IncrementalSourceStreamFetcher.shouldEmit()` uses `isAfter(splitStartWatermark)` (strict greater-than). After recovery, the first event with the same LSN as `splitStartWatermark` is filtered out — even though that event was never delivered before the savepoint.
**Bug 3 — `execute()` never checks `errorHandler.getProducerThrowable()`**
`PostgresWalFetchTask.execute()` does not check `errorHandler.getProducerThrowable()` after the streaming loop returns. Debezium 1.9.8 swallows streaming failures into the error handler, so a wedged replay after restore is silently invisible to the job.
These three bugs together explain the symptom perfectly: the offset commit advances the slot past the row, the emit filter drops the row on recovery, and the error handler never surfaces the silent gap. This is consistent with @davidzollo's evidence that two unrelated branches reproduce the failure — the bugs are in the shared CDC base path, not in any single PR's deltas.
### 2. Fix approach (PR #11864)
I adopted a **hybrid A+B approach**:
- **A (LSN key fix):** Added `getLsnCommit()` to `LsnOffset` that reads from `LAST_COMMIT_LSN_KEY` with fallback to `LSN_KEY`, and updated `commitCurrentOffset` to use the committed LSN. Also fixed the `LsnOffset` constructor to store `LAST_COMMIT_LSN_KEY` (it was missing).
- **B (emit boundary fix):** Changed `shouldEmit` from `isAfter` to `isAtOrAfter` so the watermark-position event is included on recovery.
- **Error handling:** Added `errorHandler.getProducerThrowable()` check in `execute()` to surface streaming failures.
### 3. Tests
- **`LsnOffsetTest`**: 4 deterministic unit tests covering `getLsnCommit()` fallback logic and verifying that the constructor stores `LAST_COMMIT_LSN_KEY`.
- **`IncrementalSourceStreamFetcherTest`**: 5 boundary tests covering the `isAtOrAfter`/`isAfter` comparison, including the exact watermark-position case.
All tests pass. The full diff is in #11864 — happy to adjust the approach based on your feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][Connector-V2][Postgres-CDC] Committed-offset restore can silently skip a row committed after savepoint (slot LSN advances past it) [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on issue #11847:
URL: https://github.com/apache/seatunnel/issues/11847#issuecomment-6376386605
Closing this issue as completed after rechecking the original report and its resolution.
The committed-offset savepoint/restore path now commits `lsn_commit`, includes the restored boundary record with `isAtOrAfter`, and propagates producer failures. These changes address the specific savepoint loss window reported here.
Resolution evidence:
- [PR #11864](https://github.com/apache/seatunnel/pull/11864), merged into `dev` on 2026-09-11 ([merge commit `5af8d789aa9a`](https://github.com/apache/seatunnel/commit/5af8d789aa9ae3d94df4a9cc0f03cee3c0a6d0e6)).
The corresponding implementation remains present in [`dev` at `14a27ac26a0a`](https://github.com/apache/seatunnel/commit/14a27ac26a0a71b56722f86cdf8ec2779094ceaf).
The separate snapshot/WAL backfill work in PR #11556 remains distinct and is not being closed by this action.
<!-- seatunnel-closure-audit:20261023:11847 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org