You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "davidzollo (via GitHub)" <gi...@apache.org> on 2026/07/26 03:54:27 UTC

[PR] [Fix][Connector-V2] Backfill PostgreSQL WAL during CDC snapshots [seatunnel]

davidzollo opened a new pull request, #11556:
URL: https://github.com/apache/seatunnel/pull/11556

   ## Purpose
   
   PostgreSQL/OpenGauss exactly-once initial snapshots recorded low and high watermarks but unconditionally skipped the bounded WAL backfill between them. Because the incremental split starts from the minimum high watermark, concurrent DML not observed by the snapshot query could be lost.
   
   ## Changes
   
   - Read bounded WAL from the low watermark through the high watermark for each snapshot split, and emit the END watermark only after the backfill completes.
   - Bootstrap the persistent streaming slot before the first snapshot low watermark.
   - Isolate each snapshot reader with a temporary replication slot and explicitly verify slot cleanup.
   - Preserve the full captured-table filter for pgoutput publication management while dispatching records only for the current snapshot table.
   - Generate deterministic temporary slot names within PostgreSQL's 63-character limit using the source slot hash and reader subtask.
   - Add a Zeta E2E case that performs concurrent INSERT, UPDATE, and DELETE during an active snapshot and verifies source/sink equality and slot lifecycle.
   - Add unit coverage for temporary slot naming.
   
   OpenGauss uses the shared PostgreSQL incremental source path and receives the same fix.
   
   ## Compatibility
   
   No user-facing option, default value, or public constructor is changed. The existing default of retaining the main slot is preserved, while an explicit `slot.drop.on.stop=true` is honored when the job closes. This implements the existing exactly-once contract, so no user documentation change is required.
   
   ## Validation
   
   - [x] Scoped Spotless apply and check for the affected CDC base, PostgreSQL CDC, and PostgreSQL CDC E2E modules
   - [x] Independent QA static review
   - [x] Independent maintainer static review
   - [ ] Compile, unit, and E2E execution: delegated to the current-head GitHub CI according to the Apache SeaTunnel repository workflow
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Backfill PostgreSQL WAL during CDC snapshots [seatunnel]

Posted by "DanielLeens (via GitHub) via commits" <co...@seatunnel.apache.org>.
DanielLeens commented on PR #11556:
URL: https://github.com/apache/seatunnel/pull/11556#issuecomment-6970992044

   Superseded by #13310, resubmitted under DanielLeens with the same head commit and unchanged code. The outstanding P1 review finding and original review history are explicitly carried forward; this closure does not indicate that the blocker has been fixed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Backfill PostgreSQL WAL during CDC snapshots [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11556:
URL: https://github.com/apache/seatunnel/pull/11556#issuecomment-5136841541

   本轮根据 CI 失败做了最小修复:
   
   - `PostgresCDCIT.testPostgresCdcBackfillsConcurrentChangesDuringSnapshot` 在 GitHub runner 上卡在 Zeta job startup / table discovery,日志里持续 `createdJobCount=0`,slot 还没来得及创建就耗尽 180s;把 PostgreSQL CDC 控制面等待预算调到 300s。
   - `testPostgresCdcCheckDataWithCustomPrimaryKey` 仍然使用硬编码 60s snapshot/stream wait,已改为复用同一个 `POSTGRES_CDC_WAIT_TIMEOUT_SECONDS`。
   
   本地只按 SeaTunnel 当前任务约束跑了 scoped Spotless:
   
   `./mvnw -nsu -pl seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-postgres-e2e -DskipTests -DskipIT=true -Dmaven.gitcommitid.skip=true spotless:apply spotless:check`
   
   未运行本地 compile/test/E2E。新提交已 push,等待 GitHub CI 重新验证。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Backfill PostgreSQL WAL during CDC snapshots [seatunnel]

Posted by "davidzollo (via GitHub) via commits" <co...@seatunnel.apache.org>.
davidzollo closed pull request #11556: [Fix][Connector-V2] Backfill PostgreSQL WAL during CDC snapshots
URL: https://github.com/apache/seatunnel/pull/11556


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Backfill PostgreSQL WAL during CDC snapshots [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11556:
URL: https://github.com/apache/seatunnel/pull/11556#issuecomment-5369032586

   @SEZ9 thanks for the fresh from-scratch pass — I re-checked Issue 1 against the current head (`13a093307ab5`) and it's a real, correctly-diagnosed gap that my own nine rounds missed. Confirming the call chain independently:
   
   - `SourceSplitEnumeratorTask.stateProcess()` calls `this.close()` on both `CLOSED` (normal completion) *and* `CANCELLING` (`SourceSplitEnumeratorTask.java`, the `CANCELLING: this.close(); currState = CANCELED;` branch) — so enumerator close is not a terminal-only event, it also fires on task cancellation, which is what a Zeta pipeline restart/failover drives.
   - `SourceSplitEnumeratorTask.close()` unconditionally calls `enumerator.close()`, which for the hybrid case reaches `HybridSplitAssigner.close()` → `snapshotSplitAssigner.close()` → `SnapshotSplitAssigner.close()` at `SnapshotSplitAssigner.java:277-279`, which itself unconditionally calls `dialect.closeEnumerator(sourceConfig)` — no terminal-vs-transient distinction at any layer of this chain.
   - `PostgresDialect.closeEnumerator()` (`PostgresDialect.java:174-215`) only skips the drop when `DROP_SLOT_ON_STOP` is false, or when the slot's `SlotState` is still `slotIsActive()` at that instant. It does not check whether the job is actually finishing versus being torn down for restart. During a failover the incremental readers are torn down around the same time as the enumerator, so there's a real window where the slot shows inactive and gets dropped — exactly the scenario you described, and it does directly threaten resume-from-checkpoint (the persistent slot's `confirmed_flush_lsn` is what the subsequent incremental split resumes from).
   
   So I'm updating my own conclusion: this needs to be added as a blocker alongside the CI-not-yet-green and #11847/#11864 items from my last round, not treated as a nice-to-have. Your suggested fix (gate the drop on a terminal condition, e.g. only from `SnapshotOnlySplitAssigner`/completed state, never from the `HybridSplitAssigner` snapshot leg while incremental is still the live path) looks like the right shape to me too.
   
   On the rest of the list: Issues 2 and 6 read as the same underlying finding (double-invoke of `closeEnumerator` on the `open()` failure path plus the framework's own `close()`) — worth folding into one fix. Issues 3–5 and 7 (slot-name charset validation, deterministic-name collision on crash/restart, byte- vs char-length truncation, and the `max_replication_slots` sizing note in the FAQ) all check out against the code as legitimate hardening gaps for exactly-once correctness, even if lower severity than Issue 1. Issue 8's try/finally point on `HybridSplitAssigner.close()` is a straightforward correctness fix.
   
   @davidzollo — given Issue 1 is a genuine data-loss-on-failover risk in code that's been stable across nine review rounds, I'd treat it as a must-fix alongside the two carried-forward blockers before this merges.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Backfill PostgreSQL WAL during CDC snapshots [seatunnel]

Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx commented on code in PR #11556:
URL: https://github.com/apache/seatunnel/pull/11556#discussion_r4374264274


##########
seatunnel-connectors-v2/connector-cdc/connector-cdc-postgres/src/main/java/org/apache/seatunnel/connectors/seatunnel/cdc/postgres/source/reader/wal/PostgresWalFetchTask.java:
##########
@@ -118,6 +131,93 @@ public SourceSplitBase getSplit() {
         return split;
     }
 
+    /**
+     * A bounded WAL reader used to reconcile changes made while a snapshot split is scanned.
+     *
+     * <p>The reader emits an END watermark after completely processing the high watermark.
+     */
+    public static class PostgresWalSplitReadTask extends PostgresStreamingChangeEventSource {
+
+        /**
+         * Defines the inclusive low watermark and inclusive high watermark for this reader.
+         *
+         * <p>The Debezium stopping LSN is populated from this split.
+         */
+        private final IncrementalSplit walSplit;
+
+        /**
+         * Emits the END watermark into the same queue as snapshot and WAL records.
+         *
+         * <p>The scan fetcher uses that watermark to finish its reconciliation buffer.
+         */
+        private final JdbcSourceEventDispatcher<PostgresPartition> dispatcher;
+
+        /**
+         * Propagates failures captured internally by Debezium's streaming source.
+         *
+         * <p>Debezium records producer failures instead of throwing them directly.
+         */
+        private final PostgresErrorHandler errorHandler;
+
+        /**
+         * Creates a WAL reader that stops at the snapshot split's high watermark.
+         *
+         * <p>The supplied replication connection belongs only to the current snapshot reader.
+         */
+        public PostgresWalSplitReadTask(
+                PostgresConnectorConfig connectorConfig,
+                Snapshotter snapshotter,
+                PostgresConnection connection,
+                PostgresEventDispatcher<TableId> pgEventDispatcher,
+                JdbcSourceEventDispatcher<PostgresPartition> dispatcher,
+                PostgresErrorHandler errorHandler,
+                PostgresSchema schema,
+                PostgresTaskContext taskContext,
+                ReplicationConnection replicationConnection,
+                IncrementalSplit walSplit) {
+            super(
+                    connectorConfig,
+                    snapshotter,
+                    connection,
+                    pgEventDispatcher,
+                    errorHandler,
+                    Clock.SYSTEM,
+                    schema,
+                    taskContext,
+                    replicationConnection);
+            this.walSplit = walSplit;
+            this.dispatcher = dispatcher;
+            this.errorHandler = errorHandler;
+        }
+
+        /**
+         * Reads from the low watermark until Debezium completely processes the high watermark, then
+         * emits END so the snapshot fetcher can merge its buffered records.
+         */
+        @Override
+        public void execute(
+                ChangeEventSourceContext context,
+                PostgresPartition partition,
+                PostgresOffsetContext offsetContext)
+                throws InterruptedException {
+            offsetContext.setStreamingStoppingLsn(((LsnOffset) walSplit.getStopOffset()).getLsn());

Review Comment:
   [P1] Complete bounded backfill when the source database has no decodable events
   
   With `exactly_once=true` and `startup.mode=initial`, the source database can remain idle throughout the snapshot while another database on the same PostgreSQL instance commits transactions. `PostgresUtils.currentLsn()` reads the cluster-wide `pg_current_wal_lsn()`, so those commits can make `HIGH > LOW` and enter this backfill path. However, the source database's logical replication slot does not decode the other database's transactions.
   
   Passing this HIGH to `setStreamingStoppingLsn()` makes the reader wait for `lastCompletelyProcessedLsn >= HIGH`, but that position advances only when a message is decoded. If no source-database message arrives, `super.execute()` does not return and END is never emitted. `IncrementalSourceScanFetcher` then cannot deliver the buffered snapshot or complete the split, and the replication slots can continue retaining WAL.
   
   Please use a reliable backfill boundary protocol that can confirm completion across WAL that produces no decodable source-database event, while preserving transaction completeness. Add a regression test where only another database generates WAL during the snapshot and the captured database remains idle. Simply timing out and emitting END would risk reintroducing data loss.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Backfill PostgreSQL WAL during CDC snapshots [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11556:
URL: https://github.com/apache/seatunnel/pull/11556#issuecomment-5601184780

   Thanks for the independent pass on `ba7c8bd743c8`, @SEZ9 — this is a genuinely valuable second read and I don't want it to sit un-triaged against my own `2026-09-09T00:46:19Z` review, so I pulled the actual source at this head (via the contents API, not just re-reading the diff) to verify the three findings with the highest correctness stakes before responding.
   
   **Issue 3 (regex `table.include.list` from raw `TableId`) — confirmed, and I agree this is the most serious item in your list.** `PostgresSourceFetchTaskContext.createConnectorConfig()` at this head does exactly `.with("table.include.list", tableId.schema() + "." + tableId.table())` with no `Pattern.quote()` or equivalent escaping, and Debezium's `RelationalTableFilters` does compile that value as a regex. A table like `order$archive` would have its backfill silently filtered to zero rows, and per the runtime chain you traced, `PostgresSnapshotFetchTask` would still reach the stop LSN and dispatch `END` normally — a "successful" split that quietly drops the exact concurrent-DML window this PR exists to capture. That's not a hardening nit, it's a correctness gap in the PR's own core promise, and I'm elevating it to a blocker alongside your Issue 1.
   
   **Issue 1 (`debezium.slot.name` bypasses the new charset validation) — confirmed.** I checked `PostgresSourceConfigFactory.fromReadonlyConfig`/`create()` directly: `checkArgument(SLOT_NAME_PATTERN...)` runs against `this.slotName`, then `create()` does `props.setProperty("slot.name", slotName)` followed by `if (dbzProperties != null) props.putAll(dbzProperties)` — and unlike `include.schema.changes`, which is deliberately re-applied after that `putAll` to stay authoritative, `slot.name` is not. So a `debezium.slot.name` pass-through does silently override the validated value, and `PostgresSourceConfig.getSlotNameForBackfillTask()`'s "guaranteed single-byte ASCII" Javadoc claim no longer holds for that path. Given my own Issue 1 from the last round (missing test coverage for the *happy*-path validation) is really the same seam from a different angle, I'd fold both into one ask: validate the effective post-merge `props.getProperty("slot.name")`, not just the pre-merge field, and
  add tests for both the accept/reject cases and the bypass case.
   
   **Issue 2 (`closeEnumerator` unreachable for the documented FAQ behavior) — confirmed.** `HybridSplitAssigner`'s two constructors both hardcode `false` for `releasesEnumeratorResourcesOnCompletion` (with a comment explaining the incremental phase's dependency, which is the right call for the flag's value) — I checked both constructors directly. Since `IncrementalSource` only builds `SnapshotOnlySplitAssigner` (the one that passes `true`) for `StartupMode.SNAPSHOT_ONLY`, the only mode the FAQ text describes (`exactly_once && INITIAL`) never takes the assigner that would call `closeEnumerator` on a normal completion. This is a docs/Javadoc-vs-code mismatch, not a data-safety bug, so Medium is the right severity — but it should block merge alongside Issues 1/3 since it's describing a guarantee to operators that doesn't exist.
   
   I haven't independently re-verified Issues 4/5/6 line-by-line the way I did for 1/2/3, but the evidence you've quoted is specific and consistent with what I've already confirmed elsewhere in this file (the `DROP_SLOT_ON_STOP=false` config is visible in the same `createConnectorConfig`/`createReplicationConnectorConfig` methods I just checked for Issue 3), so I have no reason to doubt them and I'm not going to make you re-litigate them.
   
   **Updated blocker list for this round, superseding the "test coverage only" framing in my last review:**
   1. Your Issue 3 — escape/quote the per-split `table.include.list`, or scope the backfill dispatcher with an equality-based filter instead of the regex-based include list. This is the one that can silently reintroduce the bug the PR fixes.
   2. Your Issue 1 — validate the effective (post-`dbzProperties`-merge) `slot.name`, not just the pre-merge field; my own prior "no test coverage" finding folds into this as the same fix's test.
   3. Your Issue 2 — either fix the four docs pages + the `closeEnumerator` Javadoc to describe what actually happens (nothing, on a normal `INITIAL` shutdown), or wire a reachable enumerator-side drop for the hybrid path.
   4. Carried over from my last review: get a clean CI run (sync `dev` first — it already has the `DorisIT.testCustomSql` fix that's the likely cause of the current `doris-connector-it` failure).
   
   Issues 4/5/6 and my own carried-over Issue 3 (JDBC-connection-per-poll in the test helper) stay non-blocking recommended fixes, same as before.
   
   @davidzollo — sorry for the extra round of findings this late in the review; Issue 3 in particular is worth prioritizing since it's the one with real data-loss potential, and it's a small, self-contained fix (`Pattern.quote(...)` at the two `table.include.list` call sites) that shouldn't need another design discussion.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Backfill PostgreSQL WAL during CDC snapshots [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11556:
URL: https://github.com/apache/seatunnel/pull/11556#issuecomment-6343964772

   The current head `dbc53649a3c17794d692ecc972db54a12401ace8` is ready for maintainer re-review.
   
   - Synced with the latest `dev` (`4dcfa28ccc85837edf8ca08e459ae56ddc1b34fd`): behind 0, mergeable, no conflicts.
   - Exact-head fork Build completed successfully: 76 jobs total, 43 successful, 33 conditionally skipped, 0 failed/cancelled/timed out. Run: https://github.com/davidzollo/seatunnel/actions/runs/39766030901
   - PostgreSQL CDC and OpenGauss CDC integration matrices passed on JDK 11 and 17, including the concurrent snapshot-time INSERT/UPDATE/DELETE backfill coverage.
   - Kafka, Redis, MySQL CDC, Elasticsearch, Doris, Oracle CDC, Iceberg, and all four unit-test matrices are green on this head.
   - The latest full source review found no source-side blockers and concluded **Ready to merge**: https://github.com/apache/seatunnel/pull/11556#pullrequestreview-5614463819
   - No unresolved review threads remain.
   
   The only remaining repository gate is a maintainer approval / merge-queue action. The upstream `labeler` and `Notify test workflow` jobs are still queued, but the `dev` ruleset requires only the `Build` context plus one approval; the exact-head Build evidence above is green.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Backfill PostgreSQL WAL during CDC snapshots [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11556:
URL: https://github.com/apache/seatunnel/pull/11556#issuecomment-5579479463

   Resolved the merge conflict against `dev` (one file, `PostgresSourceFetchTaskContext.java` — two independent additive hunks: this PR's legacy 3-arg constructor and `close()` replication-connection cleanup vs. `dev`'s new `lastRelationSchemas`/`relationSchemaBaseline` fields and schema-listener cleanup; kept both sides) and pushed `ba7c8bd743c`, which also closes out the non-blocking follow-ups from your last review round on top of the Issue 1 fix already on this branch:
   
   - **Issue 8** — `HybridSplitAssigner.close()` now uses try/finally so a failure closing the snapshot assigner can't skip closing the incremental assigner.
   - **Issues 2/6** — `SnapshotSplitAssigner` now tracks whether `openEnumerator` has an outstanding, unclosed call (`dialectOpened`). `open()`'s catch-block cleanup and `close()`'s completion-gated cleanup share that guard, so `closeEnumerator` can't fire twice for one `openEnumerator` — closing the "slot does not exist on the second call" masking risk. `dialect.openEnumerator(sourceConfig)` moved inside `open()`'s try block so a partial failure there is also covered by the catch-side cleanup. Updated the three existing `close()` gating tests to call `open()` first (matching real lifecycle usage now that `close()` only releases what `open()` actually acquired) and added a dedicated regression test for the double-close fix.
   - **Issue 3 (and 5, which it subsumes)** — `PostgresSourceConfigFactory` now validates `slot.name` against `^[a-z0-9_]{1,63}$` at config-factory time, failing fast at job submission. Since that guarantees the name is single-byte ASCII, the existing char-count truncation in `PostgresSourceConfig` is now provably byte-accurate rather than resting on an unenforced assumption — documented on both call sites instead of touching the truncation logic itself. I did not add a dedicated unit test for the new validation call path itself (no existing test in this file exercises `fromReadonlyConfig`, and building a minimal-but-correct `ReadonlyConfig` for it blind felt riskier than the one-line guard clause it's protecting); flagging that gap rather than shipping an unverified test.
   - **Issue 7** — added the `max_replication_slots >= parallelism + 1` sizing note to all four PostgreSQL/OpenGauss EN/ZH docs.
   
   **Issue 4 not addressed** — the deterministic backfill-slot-name collision after a reader crash. A correct fix needs a drop-if-exists-with-retry (or per-attempt naming) change inside the path that builds Debezium's replication connection for the backfill slot, which I don't think I should design and ship without a real PostgreSQL failure-injection test bed to verify the retry/backoff behavior against. Given it's rated Medium/non-blocking, I'd rather flag it here than guess.
   
   `./mvnw spotless:apply` run on `connector-cdc-base` and `connector-cdc-postgres` before pushing, no further formatting changes produced. Per the SeaTunnel local-verification rule I'm not running the modules' tests locally — GitHub CI on this head is the verification source of truth.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Backfill PostgreSQL WAL during CDC snapshots [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11556:
URL: https://github.com/apache/seatunnel/pull/11556#issuecomment-6338939037

   Synced this PR with the latest `dev` (`4dcfa28ccc8`) in merge commit `dbc53649a3c`; the merge was clean.
   
   The sync brings in both upstream fixes observed in the previous head CI:
   - #12938 / `cc54bb43335` fixes the unrelated Transform legacy-wrapper deserialization failures that affected all four unit-test matrix jobs.
   - #12946 / `4dcfa28ccc8` raises the MySQL CDC integration-test timeout from 120 to 180 minutes; the previous JDK 17 job was cancelled exactly at the old two-hour limit while its tests were still passing, while the JDK 11 counterpart succeeded.
   
   Relevant connector evidence from the previous head (`00337a9e`):
   - PostgreSQL updated-modules part-2 passed on JDK 11 and JDK 17.
   - OpenGauss updated-modules part-4 passed on JDK 11 and JDK 17. `OpengaussCDCIT` was 23/23 with zero failures on both, including all five custom-primary-key invocations.
   
   Local verification after the merge:
   
   `./mvnw -nsu -pl seatunnel-connectors-v2/connector-cdc/connector-cdc-base,seatunnel-connectors-v2/connector-cdc/connector-cdc-postgres,seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-postgres-e2e,seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-opengauss-e2e -DskipTests -DskipIT=true -Dmaven.gitcommitid.skip=true spotless:apply spotless:check`
   
   Result: `BUILD SUCCESS`; `git diff --check` is clean. Per the repository local-validation boundary, I did not run local compile/package/E2E. A fresh GitHub Build for `dbc53649a3c` is queued: https://github.com/davidzollo/seatunnel/actions/runs/39766030901


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Backfill PostgreSQL WAL during CDC snapshots [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11556:
URL: https://github.com/apache/seatunnel/pull/11556#issuecomment-6411602805

   Thanks @nzw921rx for the careful read, and sorry for missing this in my last review. I re-traced the bounded backfill on the current head `dbc53649a3` (source read only, nothing was executed locally), and I agree with the P1 finding. My earlier "Ready to merge" conclusion no longer holds, so please treat it as withdrawn until this is resolved.
   
   This is how I confirmed it:
   
   ```text
   PostgresSnapshotFetchTask.execute
     -> changed = HIGH.isAfter(LOW)                      (the only gate before the WAL backfill)
        HIGH and LOW come from PostgresUtils.currentLsn -> currentXLogLocation, the cluster-wide WAL position
     -> PostgresWalSplitReadTask.execute                 (PostgresWalFetchTask.java:203)
        -> offsetContext.setStreamingStoppingLsn(HIGH)
        -> PostgresStreamingChangeEventSource.processMessages  (loop condition around line 269)
           loops while lastCompletelyProcessedLsn < stoppingLsn
           lastCompletelyProcessedLsn is only assigned inside the readPending callback, when a decoded message with isLastEventForLsn arrives
           the no-message branch (around line 379) only dispatches a heartbeat and pauses; it never advances that LSN
     -> END watermark is dispatched only after super.execute() returns
   ```
   
   So if the cluster WAL position moves from LOW to HIGH but the source database's slot never decodes a message at or after HIGH (for example, commits only in another database on the same instance), `super.execute()` keeps polling and END is never emitted. The buffered snapshot rows are then never merged, and the split cannot complete. The slot also keeps retaining WAL while it waits.
   
   The comment this PR already adds in `PostgresUtils.currentLsn` (about `txid_current()` creating an artificial gap that makes the backfill wait forever) describes the same failure mechanism. That change closes one source of undecodable gaps, but the boundary is still cluster-wide, so other sources of undecodable WAL can still produce `HIGH > LOW`. Inference, not verified here: periodic WAL records and WAL written for tables the slot filters out could produce the same gap even without a second database. I have not confirmed that on a live instance, so please treat it as a question to check while fixing this.
   
   Also, the new ITs in `PostgresCDCIT` generate concurrent INSERT/UPDATE/DELETE on the captured table, so a decodable message always lands at or after HIGH. I could not find a case where the captured database is idle while other WAL is produced, which is why CI was green.
   
   I agree with the requested direction and with the caution that a plain timeout followed by END could bring back the data loss this PR is fixing. A few options to evaluate (suggestions, not verified against this code):
   
   1. Make the HIGH boundary decodable from the source database: after reading HIGH, emit a marker from the captured database that the slot is guaranteed to decode (for example `pg_logical_emit_message` with the transactional flag, or a write to a small watermark table), and use the LSN of that marker as the stopping position. The marker must come after every snapshot-visible change and still be read under the existing transaction-completeness rules.
   2. Alternatively, derive the stop condition from the slot's confirmed or received position (keepalive-advanced `lastReceivedLsn`), but only if it can be shown that this cannot skip an in-flight transaction that commits below HIGH.
   
   Whichever protocol is chosen, a regression test is needed: an exactly-once initial-snapshot job on `PostgresCDCIT`/`OpengaussCDCIT` where only a second database generates WAL during the snapshot and the captured database stays idle, asserting that END is emitted and the job finishes without leaving the backfill slot behind. This should fail on the current head.
   
   Everything else from my previous review is unchanged: the two Medium follow-ups (per-split backfill slot create/drop cost, and the deterministic backfill slot name after a reader crash) stay non-blocking. My updated view is "Ready to merge after fixes", with this P1 as the only blocker. I will re-review once a new commit is pushed.
   
   One process note: my earlier approval is still shown on the PR but is superseded by this comment. I do not have write-level review controls, so a committer may want to keep the PR out of the merge queue until the P1 is resolved.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Backfill PostgreSQL WAL during CDC snapshots [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11556:
URL: https://github.com/apache/seatunnel/pull/11556#issuecomment-5134596926

   I pushed a small follow-up to address the current CI failures Daniel called out on head 85e5ee1e5393.\n\nWhat changed:\n- Extended the new PostgreSQL backfill E2E control-plane waits from the fixed 30s window to a named 180s CDC wait budget. The failed CI log showed the job/table discovery consumed the original window and the job only finished submitting right as the assertion timed out.\n- Extended the PostgreSQL custom-primary-key snapshot/stream equality waits from 60s to the same named CDC wait budget.\n- Extended the matching OpenGauss custom-primary-key equality waits too, because the same Build run failed the OpenGauss CDC custom-primary-key cases in the shared CDC path.\n\nThe assertions are unchanged: the tests still require the backfill slot to be observable and still require exact source/sink equality. This only fixes synchronization under loaded CI runners.\n\nLocal validation, within the SeaTunnel boundary requested here:\n\n```\n./mvnw -pl seatunnel-e2e/seatunnel-con
 nector-v2-e2e/connector-cdc-postgres-e2e,seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-opengauss-e2e spotless:apply spotless:check -DskipTests -DskipIT=true -Dmaven.gitcommitid.skip=true\n```\n\nResult: BUILD SUCCESS. I did not run local compile/test/package/E2E/Docker/services.\n\nRemote readback after push:\n- Head: 5f09c18218d4c0e4a5afd70410a6cd066a360485\n- Mergeable: MERGEABLE\n- Merge state: BLOCKED\n- Review gate: REVIEW_REQUIRED until re-review\n- CI: new checks are queued/running on the pushed head.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Backfill PostgreSQL WAL during CDC snapshots [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11556:
URL: https://github.com/apache/seatunnel/pull/11556#issuecomment-5137717392

   本轮继续推进到可交付状态,已根据当前 head 的 GitHub CI 失败点做最小修复并推送:
   
   Pushed commit: 43cf72d49f1414e715b5595be9362c8fece1e8f5
   
   CI 失败归因:新的 Build 在 unit-test (11, windows-latest) 中失败,失败模块是 seatunnel-edge-agent-connector,具体用例 FileCollectReaderBehaviorTest.rediscoversFileAfterInactiveCursorClosed 在 Windows runner 上 3 秒内没有等到文件重新发现后的第二条事件。
   
   本次修复:没有降低断言、没有跳过测试,只把该文件系统轮询用例的 rediscovery Awaitility 预算抽成 FILE_REDISCOVERY_TIMEOUT_SECONDS 并从 3s 调整为 10s,保留必须读到 exactly one event 且 payload 只包含 second、不包含 first 的严格断言。
   
   本地验证边界:按 Apache SeaTunnel 本地规则,本轮只执行 scoped Spotless:
   ./mvnw -nsu -pl seatunnel-edge-agent/seatunnel-edge-agent-connector -DskipTests -DskipIT=true -Dmaven.gitcommitid.skip=true spotless:apply spotless:check
   结果:BUILD SUCCESS。
   
   当前功能验证继续以新 head 的 GitHub CI 为准。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Backfill PostgreSQL WAL during CDC snapshots [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11556:
URL: https://github.com/apache/seatunnel/pull/11556#issuecomment-5140505575

   Fixed the public API compatibility concern raised in the latest review.
   
   What changed:
   - Restored the legacy public constructor signature for PostgresSourceFetchTaskContext:
     PostgresSourceFetchTaskContext(JdbcSourceConfig, JdbcDataSourceDialect, PostgresConnection)
   - The restored constructor delegates to the new schema-history-aware constructor with an empty history list, so existing external callers remain source-compatible while the PR's dialect path still passes split-specific table history.
   
   Local verification:
   - Ran ./mvnw -pl seatunnel-connectors-v2/connector-cdc/connector-cdc-postgres spotless:apply
   - Per the SeaTunnel local verification rule for this task, I did not run local compile/test/package/E2E; GitHub CI remains the source of truth.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Backfill PostgreSQL WAL during CDC snapshots [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11556:
URL: https://github.com/apache/seatunnel/pull/11556#issuecomment-5611785377

   @DanielLeens thanks for verifying these against the source at `ba7c8bd743c8` rather than just the diff — I agree with your triage on all three.
   
   **Issue 3 (regex `table.include.list` from raw `TableId`)** — agreed on elevating this to a blocker. A backfill for a table like `order$archive` is filtered to zero rows, yet the split still reaches the stop LSN and dispatches `END`, so the data loss is invisible. Concrete ask: escape the per-split value in `PostgresSourceFetchTaskContext.createConnectorConfig()` (e.g. `Pattern.quote()` on schema and table separately, or an equivalent Debezium-safe form), and add a test with a table name containing a regex metacharacter that asserts the backfill actually captures the concurrent DML.
   
   **Issue 1 (`debezium.slot.name` bypass)** — agreed, and folding the happy-path coverage ask into the same seam makes sense. Concrete ask: run the `SLOT_NAME_PATTERN` check against the effective `slot.name` after the `putAll(dbzProperties)` merge, the same way `include.schema.changes` is re-applied to stay authoritative; then add tests for (a) a valid `slot.name` accepted, (b) an invalid `slot.name` rejected, and (c) `debezium.slot.name` carrying an invalid value being rejected rather than silently overriding. That also restores the `getSlotNameForBackfillTask()` Javadoc guarantee, since both the backfill slot name and the `pg_drop_replication_slot('...')` concatenation derive from that effective value.
   
   **Issue 2 (`closeEnumerator` unreachable for the documented FAQ behavior)** — agreed. Since the `HybridSplitAssigner` constructors hardcode `false` for `releasesEnumeratorResourcesOnCompletion`, the FAQ text describing the enumerator dropping the streaming slot on `slot.drop.on.stop=true` for exactly-once initial jobs doesn't match what runs. Either the docs should describe the actual behavior or the code path should be made reachable — I'd lean toward fixing the docs in this PR and leaving the behavior change for a follow-up unless the author wants to take it on here.
   
   Still open from my side, but not blockers: backfill slot persistence after a reader crash (persistent `*_st_backfill_*` slots retaining WAL), the `TXID_KEY` change in `PostgresUtils.currentLsn` not being called out in the PR description or docs, and the FAQ showing bare `slot.drop.on.stop` rather than `debezium { slot.drop.on.stop = true }`.
   
   Summary of what's blocking from my perspective: Issue 3 fix + test, and Issue 1 post-merge validation + the three tests above. Happy to re-review once a new head is pushed.
   
   <!-- streview-comment:933 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Backfill PostgreSQL WAL during CDC snapshots [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11556:
URL: https://github.com/apache/seatunnel/pull/11556#issuecomment-5612001617

   @SEZ9 Agreed, we're aligned — Issue 3 (regex `table.include.list`) and Issue 1 (post-merge `slot.name` validation) are the two real blockers, Issue 2 is the docs-vs-code mismatch that should land alongside them, and 4/5/6 plus the three items you just added (backfill-slot persistence after a reader crash, the undocumented `TXID_KEY` change, and the FAQ config-path formatting) stay non-blocking recommended fixes — nothing there changes my triage.
   
   @davidzollo to keep this simple to act on, the two blockers together are small, localized fixes: `Pattern.quote()` (or equivalent) on the per-split `table.include.list` value in `PostgresSourceFetchTaskContext.createConnectorConfig()`, and re-validating `slot.name` against `SLOT_NAME_PATTERN` after the `dbzProperties` merge in `PostgresSourceConfigFactory` the same way `include.schema.changes` is already re-applied post-merge. Once those land with the tests SEZ9 and I both asked for, and CI is green off a synced `dev`, I'm ready to re-review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Backfill PostgreSQL WAL during CDC snapshots [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11556:
URL: https://github.com/apache/seatunnel/pull/11556#issuecomment-5645089840

   Thanks for the continued work here. Two things together: (1) GitHub is currently reporting merge conflicts between this branch and the latest `dev` -- could you please sync with `dev` and resolve those first? (2) Separately, the two correctness blockers @SEZ9 and I aligned on are still open: escaping/quoting the per-split `table.include.list` value in `PostgresSourceFetchTaskContext.createConnectorConfig()`, and re-validating `slot.name` against `SLOT_NAME_PATTERN` after the `dbzProperties` merge in `PostgresSourceConfigFactory`. Once the conflicts are resolved, those two fixes (plus tests) land, and CI is green off the synced `dev`, I'm ready to re-review.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Backfill PostgreSQL WAL during CDC snapshots [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11556:
URL: https://github.com/apache/seatunnel/pull/11556#issuecomment-5323461794

   Diagnosis of the two `updated-modules-integration-test-part-1` failures at head `506e295e3037` (run 31962112336, identical on JDK 8 and 11) — they have different causes:
   
   **1. `testPostgresCdcBackfillsConcurrentChangesDuringSnapshot` (30s ConditionTimeout at the lock-wait assertion) — test-design bug, fixed by the commit just pushed (2512249c71c).**
   The blocking job future plus the three gate-blocked DML futures all ran on `ForkJoinPool.commonPool()`, which has only three workers on the 4-vCPU GitHub runners (parallelism = nproc - 1). Demand was 4, so the third DML never started, `countSourceChangesWaitingForTableLock >= 3` could mathematically never pass, and the test always timed out after 30s — deterministically on CI, while passing on developer machines with more cores. The fix pins those futures to a dedicated 4-thread pool; no assertions changed.
   
   **2. `testPostgresCdcSnapshotOnlyAndCommittedOffsetStartupModes` (0-vs-1 rows after savepoint restore) — not caused by this PR; tracked in #11847.**
   The restored committed-offset job emitted zero rows while the slot's committed LSN advanced past the post-savepoint row's commit LSN (permanent silent loss). The test method is byte-identical to dev's, none of this PR's deltas are reachable on that path (all gated on exactly-once and/or INITIAL startup), and the same scenario passed 3/3 on this PR before the `86bcff279f6` dev merge and failed 2/2 after. Full LSN timeline and analysis in #11847. Until that is resolved, expect this one test to stay red here; a plain rerun is unlikely to help.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Backfill PostgreSQL WAL during CDC snapshots [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11556:
URL: https://github.com/apache/seatunnel/pull/11556#issuecomment-6359794293

   Thanks for the clear status update, @davidzollo. I re-checked live just now: the head is still `dbc53649a3c1` (the same head I reviewed in my last full review), and the upstream `Build` check on this exact head now reports SUCCESS, together with `labeler` and `Notify test workflow`. That was the only gate I had left open (my last review noted the fork Build was still in progress), so there is nothing further from me on the source side: my conclusion stays Ready to merge. The two Medium follow-ups from that review (per-split backfill slot create/drop cost, and the deterministic backfill slot name after a reader crash) remain non-blocking and fine to track separately.
   
   One process note: the earlier CHANGES_REQUESTED review from me is still visible on the PR. All of my technical blockers from that review are cleared, but I do not have write-level review controls, so a maintainer with write access will need to do the final approve or dismiss before merge-queue entry. The branch is mergeable with no conflicts (12 commits behind `dev` is metadata only, no sync needed). Thanks again for the persistent work on this correctness fix.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org