You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "li3zhi4 (via GitHub)" <gi...@apache.org> on 2026/08/06 15:24:39 UTC

[PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

li3zhi4 opened a new pull request, #11677:
URL: https://github.com/apache/seatunnel/pull/11677

   ## Why
   
   After a CDC job (e.g. MySQL CDC → a JDBC/Doris sink) fails and the engine restarts it from the last successful checkpoint, the source can silently stop producing data: the checkpoint completes, the job stays `RUNNING`, but no records flow. The reader-side restored splits are handed back to the enumerator via `RestoredSplitOperation` → `IncrementalSourceEnumerator.addSplitsBack()`, which only calls `splitAssigner.addSplits(splits)`. When the restored split arrives **after** the reader already sent its split request, `assignSplits()` is never re-invoked — the split stays in the assigner, the waiting reader never receives it, and the job idles forever on empty checkpoints.
   
   This is a different root cause from #10574 (Kafka source restore-offset logic, fixed by #10612): that one is in the Kafka source; this one is in the CDC enumerator's restored-split assignment.
   
   ## What changed
   
   ### `IncrementalSourceEnumerator.java`
   
   `addSplitsBack()` becomes synchronous: after adding the restored splits to the assigner, if the enumerator is already running, immediately run `assignSplits()` so the splits are dispatched to the currently waiting reader and consumption resumes from the checkpointed offset.
   
   ```java
   @Override
   public synchronized void addSplitsBack(List<SourceSplitBase> splits, int subtaskId) {
       LOG.debug("Incremental Source Enumerator adds splits back: {}", splits);
       splitAssigner.addSplits(splits);
       if (running) {
           assignSplits();
       }
   }
   ```
   
   Non-goals: no checkpoint-semantics change, no sink stream-load/transaction change, no change to the transient sink error handling.
   
   ## File changes
   
   | File | Change |
   |---|---|
   | `connector-cdc/connector-cdc-base/.../source/enumerator/IncrementalSourceEnumerator.java` | +3 (synchronized + `if (running) assignSplits()`) |
   | `connector-cdc/connector-cdc-base/src/test/.../enumerator/IncrementalSourceEnumeratorTest.java` | new test `shouldAssignRestoredSplitsToWaitingReader` |
   | `seatunnel-e2e/.../connector-cdc-mysql-e2e/.../mysql/AbstractMysqlCDCITBase.java` | new e2e `testMysqlCdcContinuesAfterSinkFailure` |
   | `seatunnel-e2e/.../connector-cdc-mysql-e2e/src/test/resources/mysqlcdc_to_mysql_with_sink_failure_recovery.conf` | new e2e conf |
   
   ## Verification
   
   - **Unit test**: `IncrementalSourceEnumeratorTest#shouldAssignRestoredSplitsToWaitingReader` — reader already waiting, restored split must be dispatched immediately.
   - **Reproduced on upstream dev (2026-08-06)**: the ported test **fails on the original code** (`expected: <[SnapshotSplit(tableId=null, ...)]> but was: <[]>` — restored split never assigned) and **passes after the fix** (`Tests run: 1, Failures: 0, Errors: 0`, `BUILD SUCCESS`).
   - **New official E2E (Zeta + Testcontainers)**: `MysqlCDCIT#testMysqlCdcContinuesAfterSinkFailure` — completes snapshot/checkpoint, asserts RUNNING + source/sink consistency, injects a JDBC sink failure via a temporary MySQL trigger, waits for Zeta auto-restart, removes the trigger, writes new binlog data, and asserts consistency again (validated locally on 2.3.13, 1/1 passed, ~95 s; will run on dev CI).
   
   ## Checklist
   
   - [x] Code change complete and verified locally on 2.3.13
   - [x] Ported onto `dev` + regression test reproduced the race before the fix and passes after it
   - [ ] `mvn spotless:check` passes
   - [ ] Related unit tests pass
   - [x] Issue(s) referenced: #11676
   - [ ] Docs updated if needed (none needed — behavior fix only)
   
   ## Note for upstream rebase
   
   `IncrementalSourceEnumerator` on `dev` lives at `connector-cdc/connector-cdc-base/.../source/enumerator/` — same package layout; the patch ports as-is. The enumerator API (`addSplitsBack`, `assignSplits`, `running`) is unchanged on dev.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-6009495354

   Thanks @li3zhi4 for the real Docker/Zeta run on `677216dd4f` — a green `testMysqlCdcContinuesAfterSinkFailure` at ~120.9s is useful, and noted that the old 30s wait has been aligned to the 2-minute budget in this commit.
   
   A few things I'd still like to pin down before marking the earlier points resolved:
   
   1. **Which three robustness changes?** Could you map them to the earlier points? In particular, does this head include (a) a guard around `getServerLogs().substring(logOffset)` so a shrunk/rotated log retries inside the awaitility loop instead of throwing `StringIndexOutOfBoundsException`, and (b) anything for the window where the restarted job can replay the failing insert before the `finally` block's `DROP TRIGGER` lands and burns extra restart attempts? A single passing run doesn't by itself show whether those paths were exercised.
   
   2. **`if (running)` gate in `addSplitsBack()`.** Splits returned before `run()` executes are assigned only via `run()`'s trailing `assignSplits()`. The E2E restart test already exercises that path, so no extra unit test is needed — but could you add a short comment in the enumerator documenting that ordering so it isn't broken accidentally later?
   
   3. **`SourceSplitEnumerator` javadoc.** The rewritten contract applies to every connector's `addSplitsBack()`, but only the CDC base enumerator was hardened. Either narrow the wording back to what the engine actually guarantees for all implementations, or state in the PR description that other enumerators are expected to follow in separate PRs.
   
   Please post the CI result for the new head here when it lands, and I'll do a final pass then.
   
   <!-- streview-comment:1559 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-6228695631

   Thanks for pushing this through — good to see the `Build` run on head `2ba8abe87` (actions run 38893273138) come back green. That clears the CI side, but I'd still like to close out the points from the earlier review before this goes to the merge queue, since none of them were addressed in the thread yet:
   
   1. **`SourceSplitEnumerator` javadoc vs. other connectors (F1/F2).** The rewritten javadoc now documents an ordering obligation for *all* enumerator implementations, but only `IncrementalSourceEnumerator` was hardened. Please either (a) narrow the javadoc so it describes the engine behaviour without imposing a new contract on every `addSplitsBack()` implementation, or (b) explicitly call out that the hardening is CDC-only and file a follow-up for the remaining connectors. Which direction do you prefer?
   
   2. **Pre-run window in `addSplitsBack()` (F3/F4).** The `if (running)` gate means splits returned before `run()` executes are only dispatched via the trailing `assignSplits()` in `run()`. That invariant should at least be stated in a comment next to the gate, and ideally covered by a unit test that calls `addSplitsBack()` before `run()` and asserts the splits are still assigned.
   
   3. **E2E timing in `AbstractMysqlCDCITBase` (F5/F6).** The 30-second wait for the injected sink failure to appear in the logs is much tighter than the 2-minute budgets elsewhere in the same test; please align it. Also, please confirm the `DROP TRIGGER` in the `finally` block is guaranteed to take effect before the restarted job replays the failing insert, so a capped retry budget can't be exhausted spuriously.
   
   4. **`getServerLogs().substring(logOffset)` (F7/F8).** This can throw `StringIndexOutOfBoundsException` if the logs shrink/rotate during restart, which aborts the awaitility wait rather than retrying. A length guard (or searching the full log for the marker) would make it robust.
   
   Once those are addressed or explicitly deferred with a follow-up, I'm happy to take another look. A committer approval is still required for merge, and since the previous change request was recorded against the older head `1684c382f`, please re-request review once the above is pushed so the state can be refreshed.
   
   <!-- streview-comment:1814 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "li3zhi4 (via GitHub)" <gi...@apache.org>.
li3zhi4 commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-5235230618

   Thanks @SEZ9 — all three pre-merge asks are addressed in the squashed head `d343a51be`:
   
   1. **Test comment added**: the e2e test now carries a comment at the top of `testMysqlCdcContinuesAfterSinkFailure` stating that with `parallelism = 1` this clean full-pipeline restart drives the pre-existing `handleSplitRequest()` dispatch path and serves as a regression guard, while the new `running == true` branch in `addSplitsBack()` is covered deterministically by `IncrementalSourceEnumeratorTest#shouldAssignRestoredSplitsToWaitingReader`.
   
   2. **History squashed**: the four commits (fix+unit test, e2e+conf, two review-fix commits) are now a single clean commit `d343a51be` (6 files, +322/-5), forced-pushed with `--force-with-lease`.
   
   3. **E2E stability**: spotless:check passes on all four affected modules, the unit test passes 1/1, and the e2e module test-compiles. I've re-triggered nothing extra locally yet — if you'd like, I can push an empty commit or rely on the CI run on this new head to confirm the state-based waits (completed-checkpoint-count and job-state transition) don't flake around the 15s `job.retry.interval` window under CI load. Please let me know if you'd prefer I do that, or if the fresh CI run on `d343a51be` suffices.
   
   Also noting: the debug-log trace of a real partial-restart interleaving remains a nice-to-have follow-up, as agreed — not holding this PR for it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "li3zhi4 (via GitHub)" <gi...@apache.org>.
li3zhi4 commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-5238891603

   Thanks @DanielLeens for the fresh review on `7fd089a71` — agreed that the fixed 3s sleep in the fault-injection step was a timing assumption that could silently turn the e2e into a no-op under CI load. Replaced it with a condition-based wait on a positive signal, per your suggestion (log-based check, same pattern the kafka e2e already uses):
   
   **Change (head `779c961f3`):** after the `UPDATE f_bigint=20000` fault injection, the test now awaits the engine logs (incrementally, from a captured offset) for the `injected sink failure` message from the trigger, up to 30s. Only after that positive signal is observed does the `finally` block drop the trigger. If the injected failure never fires, the await times out and the test fails loudly instead of passing silently. The job-status signal was not usable because on dev the job status stays `RUNNING` throughout the restart window (verified by polling during a diagnostic run).
   
   **Local verification:** `MysqlCDCIT#testMysqlCdcContinuesAfterSinkFailure` passes on the new head (1/1, ~88s, `BUILD SUCCESS`), with the full path snapshot → checkpoint → injected sink failure observed in logs → trigger dropped → Zeta restore → replay succeeds → `f_bigint=30000` injected → consistency re-asserted. spotless passes on the affected module, and `git diff --check` is clean.
   
   The production fix and unit test remain byte-for-byte unchanged from what you and @SEZ9 already traced. CI on `779c961f3` will exercise the updated e2e.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-6050945202

   Thanks for the CI summary on `677216dd4f`. Good to see `mysql-cdc-connector-it` green on both JDK 8 and 11, since that is the module covering the recovery scenario here. I take your point that the remaining reds (`Dead links`, `all-connectors-it-2/6/7`, `engine-v2-it (8)`) are the same families seen on `dev`.
   
   A green run doesn't close out the earlier review points, though, since most of them aren't things CI would surface. Could you give a short status on each so I know what is addressed vs. deliberately left as-is?
   
   1. **`SourceSplitEnumerator` javadoc** — the rewritten javadoc describes an ordering obligation that only the CDC base enumerator was updated to honor. Either narrow the wording so it stays a description of the engine-side guarantee rather than a new requirement on every `addSplitsBack()` implementation, or call out explicitly that other connectors are not yet conformant. Which direction are you taking?
   
   2. **Pre-`run()` window behind the `if (running)` gate** — splits returned via `addSplitsBack()` before `run()` executes are only re-dispatched because of `run()`'s trailing `assignSplits()` call. That invariant should at least be documented in a comment next to the gate, and ideally covered by a unit test that calls `addSplitsBack()` before `run()` and asserts the splits get assigned. Is that on your list?
   
   3. **30-second wait in `AbstractMysqlCDCITBase`** — the wait for the injected sink failure to appear in the engine logs is much tighter than the 2-minute budgets elsewhere in the same test. It passed on this head, but I'd still prefer it aligned with the other budgets to avoid a new flake source. Any objection to bumping it?
   
   4. **Retry budget / trigger race** — if the restarted job replays the failing insert before the `finally` block's DROP TRIGGER takes effect, extra restart cycles get burned. Have you confirmed the retry budget has enough headroom, or made the failure injection one-shot?
   
   5. **`getServerLogs().substring(logOffset)`** — this throws `StringIndexOutOfBoundsException` if the logs shrink or rotate during the restart window, which aborts the awaitility poll instead of retrying. A bounds check or clamping `logOffset` to the current length would make the assertion robust.
   
   Once these are addressed or explicitly deferred with a note, I'm happy to take another look.
   
   <!-- streview-comment:1588 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "li3zhi4 (via GitHub)" <gi...@apache.org>.
li3zhi4 commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-5235955189

   Following up on the local e2e run and SEZ9's stability concern — I reproduced, diagnosed, and fixed a real flakiness source in the e2e test itself, then re-ran it successfully.
   
   **Root cause found (not the production fix):** the e2e's failure injection used a MySQL `BEFORE INSERT` trigger on the sink table. After the injected sink failure, the job restarts from the last checkpoint and replays the binlog events written after it (including the `f_bigint=20000` update). On dev, the restored job's replay reached the sink while the trigger was still present, so the replayed insert failed **again** — with `job.retry.times = 1` already consumed by the first failure, the job went to `FAILED` and the test's final consistency assertion never got a chance to pass. Diagnostics confirmed: `SELECT COUNT(*) FROM information_schema.triggers` after `DROP TRIGGER` = 0, and the job status stays `RUNNING` throughout the restore window on dev (it does not briefly leave RUNNING), so the job-state signal could not be used to time the trigger removal.
   
   **Fix in the e2e (squashed into the single commit `7fd089a71`):**
   - After the `UPDATE f_bigint=20000` failure injection, wait a short fixed window (3s — covers CDC consumption of the update and the failed sink write, well below the 15s `job.retry.interval.seconds` restore delay), then `DROP TRIGGER` in the `finally` block so the post-restart binlog replay does not hit the trigger a second time.
   - Verify the trigger is actually gone via `information_schema.triggers` (count = 0) before waiting for the job to return to `RUNNING`, and only then inject `f_bigint=30000`.
   
   **Result:** `MysqlCDCIT#testMysqlCdcContinuesAfterSinkFailure` now passes locally (1/1, ~63s, `BUILD SUCCESS`) on the current head `7fd089a71`, with the full path: snapshot → checkpoint → injected sink failure → Zeta restore → CDC continues → consistency re-asserted. spotless passes on the affected module.
   
   This is a test-side fix; the production change (`IncrementalSourceEnumerator.addSplitsBack()`) is unchanged and still covered deterministically by the unit test. The fresh CI run on `7fd089a71` will exercise the updated e2e.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-5231607825

   @/private/tmp/claude-501/-Volumes-disk6-tmp-seatunnel-third-nervous-swirles-887e0f/f978b90c-4e26-438b-9ad4-2fbf1f56ae20/scratchpad/pr-batch-20260809/post_bodies/reply_11677.txt


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "li3zhi4 (via GitHub)" <gi...@apache.org>.
li3zhi4 commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-5990228571

   Ran it. `MysqlCDCIT#testMysqlCdcContinuesAfterSinkFailure` on head `677216dd4f`, real Docker/Testcontainers against a MySQL container, Zeta engine:\n\n```\nTests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 120.907 s\nBUILD SUCCESS\n```\n\nFull reactor build of the e2e module's dependencies also came back green (`connector-cdc-base`, `connector-cdc-mysql`, `engine-server`, `starter`, etc. all SUCCESS). Worth noting the 120.9s wall time \u2014 it sits comfortably inside the 2-minute injected-failure budget I aligned in this commit, and would have been marginal against the old 30s wait. So the three robustness changes are exercised by a real run, not just compiled.\n\nAlso confirming from the previous comment that nothing else is outstanding on my side for this one \u2014 CI on the new head is in flight and I'll report it here.\n


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-6116668331

   Hi @SEZ9, thanks for re-checking against the current head. I traced each point on `8b430e980` (I also posted a full re-review on this head earlier, which diffed the PR-owned files against `677216dd4` and found them identical). Pointers:
   
   **F1/F2, Javadoc scope.** `SourceSplitEnumerator.java` (the `run()` Javadoc, around lines 41-57) now reads "Methods are typically called in this order..." and says the engine only guarantees `run()` after `open()` and after every reader has registered. It states that `addSplitsBack()` may arrive before or after `run()`, and that dispatch of late splits is left to the implementation. It no longer promises a fixed, non-concurrent order, so it adds no new obligation on other connectors. The stricter behavior is documented only in the class Javadoc of `IncrementalSourceEnumerator`.
   
   **F3/F4, `if (running)` gate and pre-run window.** Comments are at both sites in `IncrementalSourceEnumerator`: the pre-`assignSplits()` block in `run()` and the block above `if (running)` in `addSplitsBack()`. `addSplitsBack()` is now `synchronized`, the same monitor as `run()` and `handleSplitRequest()`, so the `running` check cannot race with `run()` setting it.
   
   **Exactly-once.** `IncrementalSourceEnumeratorTest#shouldAssignSplitsAddedBackBeforeRunExactlyOnce` does `handleSplitRequest(0)`, then `addSplitsBack(...)`, then asserts nothing is assigned yet (no early dispatch while `running == false`). After `run()` it asserts each of `splitAssigner.restoredSplits`, `context.assignedSplits` and `context.allAssignedSplits` equals the singleton list of the restored split. Because `allAssignedSplits` is cumulative, a double assignment would fail it and a drop would fail it too.
   
   **F5-F8, `AbstractMysqlCDCITBase`.**
   - F5: the failure-surface wait is `await().atMost(2, TimeUnit.MINUTES)` (about line 265), aligned with the surrounding 2-minute budgets.
   - F7/F8: the log slice is `substring(Math.min(logOffset, serverLogs.length()))` (about line 275), so a shorter or rotated log makes the poll retry instead of throwing `StringIndexOutOfBoundsException`.
   - F6: `DROP TRIGGER` in the `finally` block is not strictly ordered before a replay, and I would not claim a hard guarantee. The test drops the trigger as soon as the failure is seen, and a restart replaying the failing insert first would consume one retry. `mysqlcdc_to_mysql_with_sink_failure_recovery.conf` sets `job.retry.times = 3` with a 15-second interval, and the 15-second wait before the first restart normally covers the drop. So the tail is bounded by retry headroom rather than eliminated. That seems an acceptable trade-off to me.
   
   One caveat: the real Docker/Zeta E2E pass the author reported was on `677216dd4`, and it has not been re-run on this merge head. The PR-owned files are identical, but Build on `8b430e980` is still waiting on a fork Actions run, which a maintainer may need to approve. I have no remaining code-level concern from my side.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "li3zhi4 (via GitHub)" <gi...@apache.org>.
li3zhi4 commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-6038620296

   CI outcome on `677216dd4f` after the failed-job rerun \u2014 all residual reds are the `dev`-inherited flake families, and notably **`mysql-cdc-connector-it` is green on this head** (both JDK 8 and 11), so the E2E module this PR's recovery scenario lives in passed on CI as well. Remaining failures after rerun: `Dead links` (upstream `README.md` DeepWiki badge), `all-connectors-it-2/6/7` (8/11), `engine-v2-it (8)` \u2014 the same families documented on the `dev` runs earlier. Nothing touching this PR's code paths.\n


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "li3zhi4 (via GitHub)" <gi...@apache.org>.
li3zhi4 commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-6073708606

   @SEZ9 @davidzollo — current head is `8b430e9808`, a clean merge of current dev `3556e91cc9` onto the reviewed implementation `677216dd4f`. The PR-owned implementation and recovery test changes are retained: scoped API Javadoc, pre-run assignment comments and unit test, 2-minute failure wait, log-length clamp and retry budget of 3. The previously reported real Docker/Zeta recovery E2E pass was on `677216dd4f`; it has not been re-run on this merge head, so I am not presenting that as fresh merged-head acceptance.
   
   The sync includes actual upstream fixes for the earlier reds: #12639 DeepWiki429, #12297 unavailable MinIO images, and #12394 the exact TaskExecutionServiceTest fixture collision observed on Windows. Some engine/Postgres tests remain failing on upstream itself; these are not being silently waived.
   
   The current Build gate reports `ACTION_REQUIRED`: no fork Actions run/check suite has appeared for the new SHA, although the push is recorded, Actions is enabled and the workflows match upstream. This is distinct from a test failure. Please review the current head and update the outstanding review state through the normal approval process; a maintainer's current approving review plus successful Build is required by dev's merge queue. I cannot request formal reviews through the API with this account.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "li3zhi4 (via GitHub)" <gi...@apache.org>.
li3zhi4 commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-5989971570

   All three pushed on head `677216dd4f`. Going with option (a) on the javadoc as you preferred.\n\n**1. `SourceSplitEnumerator` contract \u2014 scoped back to what the engine guarantees.** The `@implNote` that put the re-assignment obligation on every `addSplitsBack()` implementation is removed. What replaces it states the actual guarantee, which I verified against `SourceSplitEnumeratorTask.stateProcess()` (`run()` fires only once `restoreComplete.isDone() && readerRegisterComplete`):\n\n> \"The engine only guarantees that `run()` is invoked after `open()` and after every reader has been registered. It does not guarantee the order listed above: restored splits may be returned via `addSplitsBack(List, int)` at any point of the source lifecycle, for example after a checkpoint restore, before or after `run()`. How such late returned splits are dispatched to readers is left to the implementation.\"\n\nThe stricter handling is now documented where it actually lives \u2014 on `Incrementa
 lSourceEnumerator`'s own class javadoc, describing that it tolerates out-of-order/late `addSplitsBack()` by always queueing into the `SplitAssigner` and dispatching on the first `run()` pass, or on the next `addSplitsBack`-triggered cycle once running.\n\n**2. Ordering invariant comments + test.** Comments added at both sites \u2014 the `if (running)` gate and the trailing `assignSplits()` in `run()` \u2014 both explaining that the engine delivers restored splits before `run()` because `run()` waits for full reader registration, so anything requested meanwhile is already parked in `readersAwaitingSplit`. New test `IncrementalSourceEnumeratorTest#shouldAssignSplitsAddedBackBeforeRunExactlyOnce`: registers a reader and calls `addSplitsBack(...)` before `run()`, asserts nothing is assigned yet, then calls `run()` and asserts the split is in the assigner and was assigned exactly once (the test context now accumulates all assigned splits).\n\n**3. `AbstractMysqlCDCITBase` robustness.**\n
 - Injected-failure wait raised from 30s to 2 minutes, matching the other budgets in that test.\n- `getServerLogs().substring(logOffset)` is now clamped with `Math.min(logOffset, serverLogs.length())`, so a truncated or rotated log retries instead of throwing `StringIndexOutOfBoundsException` and aborting the awaitility wait.\n- DROP TRIGGER / replay race: took the \"tolerant of an extra restart cycle\" route \u2014 `job.retry.times` in `mysqlcdc_to_mysql_with_sink_failure_recovery.conf` restored from `1` to `3` (the engine default per `EnvCommonOptions.JOB_RETRY_TIMES`), so a restart that replays the failing binlog event before the trigger is dropped consumes one retry and recovers on the next. The existing \"trigger must be gone before replay\" awaitility assertion is kept as-is.\n\nVerification: `IncrementalSourceEnumeratorTest` 2/2, `spotless:check` clean on `seatunnel-api`, `connector-cdc-base` and the mysql-cdc e2e module, and the e2e test sources compile. I have not re-run the
  live Docker/Testcontainers `testMysqlCdcContinuesAfterSinkFailure` \u2014 say the word and I'll run it. CI on the new head is in flight and I'll post the result.\n\nNo disagreement on the javadoc scoping \u2014 (a) was the right call, and it's good that the engine-side guarantee is now stated conservatively rather than leaning on behavior only the CDC enumerator provides.\n


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "li3zhi4 (via GitHub)" <gi...@apache.org>.
li3zhi4 commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-6058745098

   Thanks \u2014 the three items are all already implemented on `677216dd4f`, and the requested CI result is now posted above.\n\n1. **`SourceSplitEnumerator` javadoc:** I took the narrowing direction (option (a)). `SourceSplitEnumerator.java:52-56` now states only the engine guarantee: `run()` follows `open()` and reader registration, but restored splits may arrive through `addSplitsBack()` before or after `run()`, and dispatch of such late splits is left to each implementation. The stronger queue-and-dispatch behavior is documented on `IncrementalSourceEnumerator`, not imposed on every connector.\n2. **Pre-`run()` window:** comments are present at both `IncrementalSourceEnumerator.java:77-83` (the trailing `run()` assignment pass) and `:105-111` (the `if (running)` path). They document why a pre-`run()` restored split is queued and then dispatched once readers are registered, and why an already-running enumerator immediately retries assignment. `shouldAssignSplitsAddedBackBeforeRun
 ExactlyOnce` also covers the invariant directly; it was retained even though the later comment said the E2E path was sufficient.\n3. **30-second wait:** the injected-failure wait is now `await().atMost(2, TimeUnit.MINUTES)` at `AbstractMysqlCDCITBase.java:227`, aligned with the surrounding budgets.\n4. **Retry/trigger race:** `mysqlcdc_to_mysql_with_sink_failure_recovery.conf:24-27` now uses `job.retry.times = 3` (with the 15-second interval), leaving room for a replayed failing insert to consume one restart before the trigger removal takes effect.\n5. **Rotated/shrunk logs:** `AbstractMysqlCDCITBase.java:275` uses `substring(Math.min(logOffset, serverLogs.length()))`, so a shorter log retries instead of throwing `StringIndexOutOfBoundsException`.\n\nThe real Docker/Zeta run was 1/1 with zero failures and zero skips, and the CI MySQL CDC connector jobs were green on both JDK 8 and 11. The residual Build reds are the same unrelated dev-side families already documented. There is no fu
 rther code change needed for these five points from my side; please do the final review on `677216dd4f` when convenient.\n


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-6210067357

   Thanks @SEZ9 for the careful follow-up, and @li3zhi4 for the quick turnarounds. I re-read the four open items against the current head `c959fd198` so the author has one place to see where each stands. This is a thread reply, not a new review: the head has not changed since my last review, and my technical conclusion there is unchanged.
   
   **Items 1 / 2 (`SourceSplitEnumerator` javadoc, F1/F2):** the javadoc was already narrowed on this head. `SourceSplitEnumerator.java:42-57` now says the engine only guarantees that `run()` is invoked after `open()` and after every reader has registered, that the listed order is "typically" followed, and that how late `addSplitsBack()` splits are dispatched "is left to the implementation". The CDC-specific behavior is described in the class javadoc of `IncrementalSourceEnumerator.java:38-48`. That is the "narrow the javadoc" option @SEZ9 said is fine, so I read these two as satisfied; @SEZ9, please confirm when you do your pass.
   
   **Items 3 / 4 (`if (running)` gate, F3/F4):** the invariant is documented where the gate is: the comment block in `IncrementalSourceEnumerator.java:101-111` explains that with `running == false` the returned splits stay queued and are dispatched by the first `assignSplits()` pass in `run()`, and `run()` carries the matching comment at `:77-83`. The "addSplitsBack before run" path is also covered by a unit test: `IncrementalSourceEnumeratorTest#shouldAssignSplitsAddedBackBeforeRunExactlyOnce` (`IncrementalSourceEnumeratorTest.java:60-80`) asserts that nothing is dispatched while `running == false` (line 73) and that the split is assigned exactly once after `run()`.
   
   **Item 5 (30-second wait, F5):** I could not find a 30-second wait for the injected sink failure on this head. The failure-observed wait in `AbstractMysqlCDCITBase.java` uses `atMost(2, TimeUnit.MINUTES)` (the Awaitility block around line 265), the same budget as the convergence waits. The only 30-second wait is the check that the trigger is gone (`information_schema.triggers` count equals 0), which is a single catalog query rather than something waiting on the engine. If you were looking at an older head, this one is already aligned; if you meant that 30s check specifically, @SEZ9, let us know and @li3zhi4 can widen it, but I do not see it as a stability risk.
   
   **Item 6 (replay before `DROP TRIGGER`, F6):** this is explained in the test comment (`AbstractMysqlCDCITBase.java`, the block above the Awaitility wait) and in `mysqlcdc_to_mysql_with_sink_failure_recovery.conf:18-26`: the trigger is dropped in `finally` as soon as the failure shows up in the engine log, and if the restart wins that race the replay fails once more and consumes one of the extra retries from `job.retry.times = 3`. I checked that the budget is large enough (3 retries, 15s interval); this is a bounded one-extra-failure window, not an open-ended one.
   
   **CI, since the facts changed since my last CI note:** the head is unchanged, `Build` is still red, and the failing jobs are still the two `CassandraIT` cases on the Flink 1.20 container, which this PR cannot reach. The fix #12756 is already on `dev`, and the branch is now 27 commits behind `dev` and diverged. @li3zhi4, the lowest-cost next step is still to sync with the latest `dev` and let CI run again; if `CassandraIT` still fails after that, please paste the new failing check link here and I will help narrow it down.
   
   I do not see any remaining source-level blocker on this head. The remaining gates are a green `Build` and the human approval from @SEZ9 / a write-enabled maintainer. Thanks again for the patience on a tricky restart-path fix!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "li3zhi4 (via GitHub)" <gi...@apache.org>.
li3zhi4 commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-5211413297

   All five review issues have been addressed in the new commit `8ce669eb6`:
   
   - **Issue 1 (High)**: e2e `Thread.sleep(12)` replaced with a condition-based await on `container.getCompletedCheckpointCount(jobId) > 0` — waits until the snapshot state is durably checkpointed instead of a hard sleep.
   - **Issue 4 (Low)**: the log-keyword readiness check is replaced with a structured signal — waits until the job state leaves `RUNNING` after the injected sink failure (observable because `job.retry.interval` is 15s).
   - **Issue 5 (Low)**: added a comment explaining why `addSplitsBack()` re-runs `assignSplits()`.
   - **Issue 2 (Medium)**: corrected the `SourceSplitEnumerator.run()` Javadoc — implementations must not rely on a strict invocation order; restored splits may arrive after the reader's split request.
   - **Issue 3 (Low)**: softened the issue/PR wording from an unrecoverable hang to a bounded stall.
   
   Verified: spotless:check passes on all three affected modules and the unit test `IncrementalSourceEnumeratorTest` passes (1/1).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "li3zhi4 (via GitHub)" <gi...@apache.org>.
li3zhi4 commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-5219707861

   Thanks for the sharpened trace — I re-verified the same engine ordering and agree with the conclusion, and I've updated the PR description's "Reachability note" to reflect it precisely.
   
   **Engine-level re-assessment (matching your Issue 3):** in a clean full-pipeline restart — the exact scenario the e2e drives with `parallelism = 1` — each reader's `restoreState()` synchronously delivers its restored splits before that same reader registers, and `run()` fires only after all readers register, so `running` is `false` when `addSplitsBack()` is invoked and the new branch is a no-op there; the split is dispatched via the pre-existing `handleSplitRequest()` reassignment. The trigger for the new branch is a partial/regional restart where the enumerator task survives (`running == true`) while an individual reader task restarts and hands its restored splits back — which is now called out explicitly in the PR description.
   
   **Test evidence (nothing claimed beyond what the tests actually prove):**
   - `IncrementalSourceEnumeratorTest#shouldAssignRestoredSplitsToWaitingReader` deterministically proves the enumerator-level contract defect independent of engine timing: with the enumerator already running and a reader parked in `readersAwaitingSplit`, `addSplitsBack()` on the old code leaves the restored split unassigned (assertion fails) and on the new code dispatches it immediately (assertion passes). This is the direct, engine-independent proof of the branch the review was concerned the e2e couldn't exercise.
   - The e2e (`testMysqlCdcContinuesAfterSinkFailure`, `parallelism = 1`) validates the full sink-failure → checkpoint-restore → CDC-resumes path end-to-end through the real engine, real MySQL CDC source, and a real injected JDBC sink failure — i.e. "recovery still works" for the complete-restart configuration, as you characterized it.
   - spotless:check passes on all affected modules; the unit test passes (1/1); the e2e module test-compiles.
   
   Both tests together cover what this fix can be held to: the branch itself is proven at the enumerator boundary, and the end-to-end recovery path is proven through the real engine. The PR description now states the reachability honestly rather than overclaiming the literal ordering as the sole mechanism.
   
   A debug-log trace of the exact interleaving on a real partial-restart run remains a worthwhile follow-up, as you suggested — happy to add it in a follow-up PR if you'd prefer it in-tree before merge.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-5233647200

   Thanks for your contribution! This is a solid addition to the project, and your attention to detail really shows. Contributors like you are what keep this community moving forward. Feel free to keep the PRs coming — we're always glad to have you here!
   
   Please ignore the following message if you have joined in the Slack~
   ```
   To facilitate focused communication among contributors, we’ve created a brand-new dedicated #contributors channel in official Slack workspace.
   Slack workspace link: https://s.apache.org/seatunnel-slack
   
   This channel is designed for technical discussions, feature design brainstorming, roadmap sync and direct communication with project core Contributors. Everyone who contributes code, docs, test cases or community ideas is warmly invited to join, share thoughts and collaborate together under The Apache Way. Welcome to join 😄
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx merged PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-6105276196

   Thanks for the follow-up. I'd like to re-check my earlier points against the current head `8b430e980` rather than rely on summaries, so a few pointers would help:
   
   **Javadoc / contract scope (F1, F2)** — Scoping the `SourceSplitEnumerator` Javadoc back to what the engine actually guarantees, and documenting the stricter re-dispatch behavior only on `IncrementalSourceEnumerator`, is the approach I'd prefer. Could you point me to the relevant hunks in `8b430e980` so I can confirm the wording no longer implies a new obligation on other connectors' `addSplitsBack()`?
   
   **`if (running)` gate and pre-run `addSplitsBack()` (F3, F4)** — The ordering-invariant comments plus the `IncrementalSourceEnumeratorTest` case covering `addSplitsBack()` → `run()` sound like they address both the documentation and coverage gaps. Could you confirm that test asserts the restored splits are dispatched exactly once (no double assignment, no drop)?
   
   **E2E points (F5–F8)** — I couldn't tell from the discussion whether these were changed. Could you confirm, for `AbstractMysqlCDCITBase`:
   1. F5: whether the 30-second wait for the injected sink failure to surface in the logs was aligned with the 2-minute budgets used elsewhere, or why 30s is safely sufficient in CI.
   2. F6: whether the `DROP TRIGGER` in the `finally` block is guaranteed to take effect before the restarted job can replay the failing insert.
   3. F7/F8: whether `getServerLogs().substring(logOffset)` is guarded against the log string being shorter than `logOffset` (e.g. clamp the offset or catch and retry inside the awaitility lambda).
   
   If those are already handled in `8b430e980`, a quick pointer is all I need and I'm happy to approve.
   
   <!-- streview-comment:1673 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-5251995058

   Thanks for the detailed diagnosis, and for rerunning the full `connector-cdc-mysql-e2e` module rather than just the one test this PR touches.
   
   I agree with your read: `MysqlCDCWithFlinkSchemaChangeIT.testMysqlCdcWithSchemaEvolutionCase` failing on `ConditionTimeout` in `assertSchemaEvolutionForAddColumns` is a known-flaky test, not a regression from this PR. That test has a DDL-vs-binlog-initialization race under loaded CI runners that shows up intermittently even on PRs that never touch schema-change code, and this PR's diff genuinely doesn't touch that file. The fact that this PR's own new test, `MysqlCDCIT#testMysqlCdcContinuesAfterSinkFailure`, passed cleanly on the same run (along with the rest of the 107-test module) is the evidence that matters for this PR specifically.
   
   From Daniel's side the only remaining item was CI turning green — the production fix and unit test have been settled for a while now. Since this is an unrelated flaky test, a rerun of the failed job should be enough to get a clean gate; no code change needed here. Feel free to retrigger it, and ping me if it comes back red with something that isn't the same schema-evolution test.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-5903258002

   Thanks for the status update and the rebase, @li3zhi4. Since the head is still `5ba2e3c518`, the points from the previous review remain open. Here is what I'd like to see before this moves forward:
   
   **`SourceSplitEnumerator` javadoc / contract**
   The rewritten javadoc documents a weaker engine-side ordering guarantee and a corresponding obligation for `addSplitsBack()` implementations, but only `IncrementalSourceEnumerator` was hardened. Please either (a) scope the wording to what the engine actually guarantees today and describe the CDC enumerator's stricter handling as implementation-specific, or (b) if a contract change for all enumerators is intended, state that explicitly in the PR description and open a follow-up for the remaining connectors. I'd prefer (a) for this PR.
   
   **`if (running)` gate and pre-run reassignment**
   Splits returned via `addSplitsBack()` before `run()` executes are only dispatched through the trailing `assignSplits()` call in `run()`. That works, but it's an implicit ordering invariant. Please add a short comment at both sites explaining why the pre-run path is safe, and extend `IncrementalSourceEnumeratorTest` with a case that calls `addSplitsBack()` before `run()` and asserts the splits are assigned once `run()` completes.
   
   **E2E robustness in `AbstractMysqlCDCITBase`**
   - Align the 30-second wait for the injected sink failure with the 2-minute budgets used elsewhere in the same test.
   - Guard `getServerLogs().substring(logOffset)` so a shorter/rotated log results in a retry (e.g. clamp the offset or treat it as "no match yet") instead of a `StringIndexOutOfBoundsException` that aborts the awaitility wait.
   - For the DROP TRIGGER / replay race: either ensure the trigger is dropped before the restart proceeds, or make the assertion tolerant of an extra restart cycle so a capped retry budget can't fail the test spuriously.
   
   Once those are pushed I'll take another look promptly. If you disagree with any of this — particularly the javadoc scoping — just say so here and we can settle it in-thread.
   
   <!-- streview-comment:1422 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "li3zhi4 (via GitHub)" <gi...@apache.org>.
li3zhi4 commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-6122658702

   Thanks @SEZ9 \u2014 all requested points are present on `8b430e9808`; exact pointers below.\n\n**F1/F2 \u2014 contract scope:** `seatunnel-api/.../SourceSplitEnumerator.java:41-57` (`run()` Javadoc) now says the engine only guarantees `run()` after `open()` and all readers are registered; `addSplitsBack()` may arrive before or after `run()`, and dispatch of late splits is left to the implementation. The former implementation obligation is gone. The stricter behavior is documented only on `IncrementalSourceEnumerator`'s class Javadoc.\n\n**F3/F4 \u2014 ordering and exactly-once coverage:** `IncrementalSourceEnumerator.java:76-84` documents the trailing `run()` assignment pass for splits returned before `run()`. `:102-111` documents both `running == false` (queue now, dispatch from `run()`) and `running == true` (immediate assignment retry) branches. `IncrementalSourceEnumeratorTest#shouldAssignSplitsAddedBackBeforeRunExactlyOnce` calls `addSplitsBack()` before `run()`, then asserts
  the split is assigned after `run()` and the test context's `allAssignedSplits` contains it exactly once; the companion running-path test covers immediate dispatch.\n\n**F5\u2013F8 \u2014 E2E:**\n- **F5:** `AbstractMysqlCDCITBase.java:227` uses `await().atMost(2, TimeUnit.MINUTES)` for the injected failure, aligned with the other budgets.\n- **F6:** `mysqlcdc_to_mysql_with_sink_failure_recovery.conf:24-27` uses `job.retry.times = 3` (15s interval), allowing a replay before trigger removal to consume one retry without exhausting the job. The existing `DROP TRIGGER IF EXISTS` is kept after the checkpoint gate; the fix deliberately uses the tolerant-extra-restart option rather than assuming the `finally` ordering is atomic.\n- **F7/F8:** `AbstractMysqlCDCITBase.java:275` uses `substring(Math.min(logOffset, serverLogs.length()))`, so a rotated/shorter log returns no match and Awaitility retries instead of throwing `StringIndexOutOfBoundsException`.\n\nThe real Docker/Zeta `MysqlCDCIT#te
 stMysqlCdcContinuesAfterSinkFailure` run was 1/1, zero failures/errors/skips (120.907s); the CI MySQL CDC connector jobs were green on both JDK 8 and 11. The only remaining gate is the repository-side Build dispatch/approval, not a source-level item. A fresh formal approval from a write-enabled maintainer is still required for the merge queue.\n


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "li3zhi4 (via GitHub)" <gi...@apache.org>.
li3zhi4 commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-6123218841

   @DanielLeens @SEZ9 — the remaining Low item in review 5479163539 is now addressed in `bedcc58b4a`: the recovery comment no longer claims that observing RUNNING means replay has completed. It states that the status wait only confirms the job is alive, and the final source-to-sink equality assertion is the convergence guard. This is a comment-only change; no assertion or runtime behavior was weakened. `git diff --check` passed; I did not present this as a new live test run.
   
   One pointer correction to my previous reply: the injected-failure **log wait** is at `AbstractMysqlCDCITBase.java:265-279`, and it uses the two-minute budget. Line 227 is the separate completed-checkpoint wait. The log-length clamp is still at line 275. The trigger is removed in `finally` at lines 280-282; retry=3 deliberately tolerates a replay arriving before removal, rather than claiming atomic ordering.
   
   The current head retains the validated production fix and all F1-F8 changes. Remote Build dispatch and a current write-enabled approval remain separate gates; they are not satisfied by this comment-only update.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "li3zhi4 (via GitHub)" <gi...@apache.org>.
li3zhi4 commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-5882570813

   Status update and a merge-readiness checklist for this PR \u2014 it's been quiet for a while, so here's the current state to make it easy to pick up:\n\n**Change (unchanged, head `5ba2e3c518`):** after a checkpoint recovery, restored splits that were left assigned to non-alive readers are now reassigned to waiting readers instead of stalling \u2014 `IncrementalSourceEnumerator` uses a new `SourceSplitEnumerator#registerWaitingReader/addSplitsBack` path, with the corresponding `SourceSplitEnumeratorTask` handling on the engine side. Tests: `IncrementalSourceEnumeratorTest` extended, plus a MySQL CDC E2E recovery scenario (`mysqlcdc_to_mysql_with_sink_failure_recovery.conf`).\n\n**Freshness:** I've just synced the branch with the latest `upstream/dev` \u2014 conflict-free. This brings in two related CDC changes that have since landed upstream: `#11271` (prune removed tables from restored incremental splits) and `#11503` (restore runtime schema from checkpoint after failover). They'r
 e complementary to this fix \u2014 this PR covers reassignment to waiting readers, not table pruning or schema restore \u2014 and the merge shows no interaction. CI is running on the new head now; I'll post results here.\n\n**Ask:** could @davidzollo or @nzw921rx (or anyone with context on CDC split lifecycle) take a look and confirm whether anything else is needed for this to move forward \u2014 or let me know if the preferred direction now differs after `#11271`/`#11503`? Happy to adapt.\n


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-5265845319

   @li3zhi4 Confirmed — that matches what I independently found across three separate review rounds (Aug 7, Aug 10 x2): `synchronized` is a modifier, not part of the Java method signature (JLS 8.4.3.6), and each round I re-grepped the repo for `extends IncrementalSourceEnumerator` / `new IncrementalSourceEnumerator(` and found zero subclasses, so there's no override site SEZ9's Issue 1 could affect. Issue 2 is equally clear-cut: this diff adds zero `Option`/config surface, so there's nothing to document.
   
   Also worth noting: the required `Build` check on this exact head (`779c961f3`) is now green — the Flink schema-evolution failure from the previous run was confirmed pre-existing and unrelated to this PR's diff.
   
   So the only thing left blocking merge at this point is procedural: the formal `CHANGES_REQUESTED` from @SEZ9 was submitted against `1684c382f` (two commits behind the current head) and hasn't been updated since SEZ9's own follow-up comment said the concerns were addressed. @SEZ9, could you submit a fresh review or update/dismiss the existing one so `reviewDecision` reflects where the discussion has actually landed? Happy to re-verify anything further if useful.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-6181488021

   Quick CI follow-up for @li3zhi4 on head `c959fd198`; this is not a new review of the code. My last review (the one tied to this head) said the required `Build` check had not started yet and only the CI gate remained. That fact has changed, so I'm updating it so the earlier guidance does not go stale.
   
   **What changed:** the fork run `li3zhi4/seatunnel` actions run 38382232755 (attempt 3) has now finished and `Build` is red. The only failing jobs are `Run / all-connectors-it-2 (11, ubuntu-latest)` and `Run / all-connectors-it-2 (17, ubuntu-latest)`. In both, the failing tests are `CassandraIT.testCassandra` (line 134) and `CassandraIT.testCassandraMultiTableSource` (line 144), in the `TestContainer` run with index `[4]`, which is the Flink 1.20 container; the Flink task manager reports `AllNodesFailedException ... NodeUnavailableException` against the Cassandra container.
   
   **My read:** this PR changes `SourceSplitEnumerator` Javadoc, `SourceSplitEnumeratorTask` (comment only), `IncrementalSourceEnumerator` and the MySQL CDC E2E, and nothing on the Cassandra or Flink translation path, so I do not see a link between the diff and these two failures. This is an inference from the diff and the log, not something I could prove. Note that the Cassandra contact-point fix from #12429 is already contained in this head, so a sync with `dev` will not change this lane (the branch is 15 commits behind `dev` and diverged, which is a fact only, not a blocker).
   
   **Suggested next step:** please re-run only the two failed jobs rather than the whole workflow. If `CassandraIT` fails again in the same `[4]` Flink container with the same `NodeUnavailableException`, please paste the new failing check link here and I'll help narrow it down against that run.
   
   My technical conclusion from the last review is unchanged: no source-level blocker on this head. The remaining gates are the green `Build` and a write-enabled maintainer approval; @SEZ9's pending items are being discussed in the thread with the author. My earlier review entries may still look different in GitHub's review summary because I am a read-only reviewer here.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-6136981670

   Thanks @li3zhi4 for the update on the remaining item from review 5479163539 and for the pointer correction.
   
   I haven't yet looked at `bedcc58b4a` or the current head myself, so I can't mark anything resolved based on the description alone. I'll go through the diff for the reworded recovery comment, the injected-failure log wait and the two-minute budget, the log-length clamp, and the `finally` trigger removal with `retry=3`. On that last point, it would help if the intent behind `retry=3` is captured in a comment next to the setting so a future reader doesn't tighten it.
   
   For F1–F4, I'll also verify on the current head how the pre-run `addSplitsBack()` window in `IncrementalSourceEnumerator` and the `SourceSplitEnumerator` javadoc wording landed. If any of those are in a different place than I expect, a short pointer would be appreciated, but I'll base the outcome on the code rather than the thread.
   
   Agreed that the Remote Build dispatch and a write-enabled approval are separate gates and aren't satisfied by a comment-only update; I'll follow up on those after reviewing the head.
   
   <!-- streview-comment:1707 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "li3zhi4 (via GitHub)" <gi...@apache.org>.
li3zhi4 commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-6165460822

   @SEZ9 — yes, the log-length guard is on current head `c959fd1989`: `AbstractMysqlCDCITBase.java:267-279`, specifically line 275, uses `substring(Math.min(logOffset, serverLogs.length()))` inside the Awaitility lambda. A shorter log therefore yields an empty suffix/no match and retries rather than throwing StringIndexOutOfBoundsException. This is defensive handling; the earlier passing run did not force log rotation, so I am not claiming that path was exercised.
   
   Immutable current-head link: https://github.com/li3zhi4/seatunnel/blob/c959fd19893a30f83ba51648462720c0915ebe4b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-mysql-e2e/src/test/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/AbstractMysqlCDCITBase.java#L265-L279
   
   The remaining current-head Build failures are the CassandraIT cases already identified; this is still a red required Build, not a merge-ready green check.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "li3zhi4 (via GitHub)" <gi...@apache.org>.
li3zhi4 commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-5214946556

   Thanks for the detailed engine-level trace — it's correct, and I've converged on it.
   
   **Issue 1 (Medium) — reachability confirmed, narrative adjusted:** I traced the same chain on the current head: a reader's `restoreState()` synchronously delivers its checkpoint-restored splits via `RestoredSplitOperation` *before* it registers, and the enumerator's `run()` only fires once all readers have registered (`SourceSplitEnumeratorTask` STARTING → RUNNING). So in a clean single-parallelism full-pipeline restart — exactly the scenario the e2e drives — `running` is typically still `false` when `addSplitsBack()` is invoked, and the literal "reader registers, then split restores late" ordering is not the dominant path on dev today. The enumerator-level defect remains real and reachable whenever `addSplitsBack()` races `run()`/`handleSplitRequest()` with `running == true` (multi-reader interleaving, retry-induced delay, scheduling variance), which the deterministic unit test proves independent of engine timing. I've updated the PR description with a "Reachability note"
  reflecting exactly this, and reframed the fix as necessary defensive hardening at the enumerator's contract boundary rather than *the* mechanism of the production stall.
   
   **Issue 2 (Low) — comment added:** `SourceSplitEnumeratorTask.receivedReader()` now carries a comment at the reader-registration gate documenting the cross-class invariant (reader restores synchronously before registering; run() fires only after all readers register) and noting that `IncrementalSourceEnumerator.addSplitsBack()` defensively re-runs assignment when that ordering does not hold.
   
   Both changes are in commit `1684c382f` (pushed). spotless:check passes on the affected module; the unit test remains green. A debug-log trace of the exact e2e interleaving on a real run remains a good follow-up as you suggested, but I believe the narrative now matches the engine behavior as traced.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "li3zhi4 (via GitHub)" <gi...@apache.org>.
li3zhi4 commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-5311101417

   @davidzollo — reaching out as a maintainer since this PR's merge is blocked purely by a stale review state, not by any outstanding code issue. Summary of where things stand:
   
   **Status of PR #11677** ([Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery):
   - **CI: fully green** on head `779c961f3` — `Build` check is `success` (verified via check-runs API; the earlier Flink schema-evolution e2e failure was confirmed pre-existing and unrelated, and a rerun passed).
   - **@DanielLeens**: "Ready to merge after fixes" — independently re-traced the production fix and unit test across multiple rounds and confirmed the only remaining item was CI green (now resolved) and the review-state update below.
   - **@SEZ9**: his formal review is `CHANGES_REQUESTED`, but it was submitted against head `1684c382f` (two commits behind the current head), and both findings behind it (possible public API change; missing config docs) don't hold against the current diff — verified with concrete evidence: `synchronized` is a modifier, not part of the Java method signature, with zero subclasses of `IncrementalSourceEnumerator` in the repo; and the diff adds zero `Option`/config surface. SEZ9 himself confirmed in a comment (Aug 9) that the concerns were addressed, but never updated the formal review state, so GitHub still reports `reviewDecision: REVIEW_REQUIRED` and blocks merge. We and @DanielLeens have both asked SEZ9 to update/dismiss it, without response since.
   
   Since the code, tests, and CI are all settled and both reviewers have substantively approved, could you help unblock the merge — e.g. dismiss the stale `CHANGES_REQUESTED` or nudge @SEZ9 to update his review? Happy to provide anything else needed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-5312555037

   Confirming the summary above matches where I left things: CI is green on the current head `779c961f3` (`Build` check completed 2026-08-10T16:19:13Z, conclusion SUCCESS, verified again just now), and my position from the last review round stands unchanged — the production fix (`IncrementalSourceEnumerator.addSplitsBack()` becoming `synchronized` plus the `running`-gated `assignSplits()` call), the deterministic unit test, and the e2e have all been settled for several rounds now, and I don't have any outstanding code-level concern.
   
   The only thing left between this PR and merge is procedural: SEZ9's `CHANGES_REQUESTED` is still formally attached to `1684c382f` (two commits behind `779c961f3`), and both findings behind it don't hold against the current diff for the reasons @li3zhi4 laid out above (`synchronized` is a modifier not part of the method signature, zero subclasses/overrides in the tree, and zero config/`Option` surface added by this diff). @davidzollo, agreed this needs a maintainer to reconcile the review state — happy to re-verify anything further on my side if it helps unblock.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "li3zhi4 (via GitHub)" <gi...@apache.org>.
li3zhi4 commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-6144041046

   Thanks @SEZ9 \u2014 the retry intent is already explicit on current head `c959fd1989`. In `mysqlcdc_to_mysql_with_sink_failure_recovery.conf:18-27`, the comment states that the injected trigger may still be present when replay starts, that replay can fail one more time, and that `job.retry.times = 3` provides the extra recovery headroom (with the configured retry interval).\n\nFor the other current-head pointers: queue-and-dispatch and both `running` branches are documented in `IncrementalSourceEnumerator.java:74-115`; the shared API contract is scoped in `SourceSplitEnumerator.java:41-58`; engine ordering is explained in `SourceSplitEnumeratorTask.java:241-247`; the running-path and exactly-once pre-run tests are `IncrementalSourceEnumeratorTest.java:41-82`; and the E2E failure wait, trigger cleanup and final convergence/cancellation checks are in `AbstractMysqlCDCITBase.java:245-289` and `:305-330`.\n\nThe two-minute waits are polling budgets; I am not presenting them as proof t
 hat injection itself takes two minutes. No source-level item remains from this list. The remaining gate is current Build/approval, not code.\n


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "li3zhi4 (via GitHub)" <gi...@apache.org>.
li3zhi4 commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-6229781525

   Thanks @SEZ9. All four items are implemented on current head `2ba8abe8701dab8971ebb1fc5ea077f3a73b9545`; I am restating the exact evidence here for the re-review, rather than treating the earlier author comments as reviewer sign-off.
   
   1. **F1/F2 — narrowed shared contract:** `SourceSplitEnumerator.java:42-57` now describes only the engine guarantees (open plus reader registration before `run()`), explicitly says the listed ordering is typical rather than a universal implementation contract, and leaves late-split dispatch to each implementation. CDC-specific stronger behavior is documented in `IncrementalSourceEnumerator.java:38-48`. Current link: https://github.com/li3zhi4/seatunnel/blob/2ba8abe8701dab8971ebb1fc5ea077f3a73b9545/seatunnel-api/src/main/java/org/apache/seatunnel/api/source/SourceSplitEnumerator.java#L42-L57
   
   2. **F3/F4 — pre-run gate:** `IncrementalSourceEnumerator.java:74-84` documents that `run()` performs the first queued assignment, while `:102-111` documents why `addSplitsBack()` does not dispatch while `running == false`. `IncrementalSourceEnumeratorTest#shouldAssignSplitsAddedBackBeforeRunExactlyOnce` (`IncrementalSourceEnumeratorTest.java:60-80`) asserts no pre-run dispatch and exactly one post-run assignment. Link: https://github.com/li3zhi4/seatunnel/blob/2ba8abe8701dab8971ebb1fc5ea077f3a73b9545/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/cdc/source/enumerator/IncrementalSourceEnumerator.java#L74-L111
   
   3. **F5/F6 — timing/replay:** the injected-failure wait is already `atMost(2, TimeUnit.MINUTES)` in `AbstractMysqlCDCITBase.java:265`, matching the other convergence waits. The separate trigger-removal catalog check is the only 30-second bound. `DROP TRIGGER IF EXISTS` runs in `finally`; it is not possible to promise an absolute ordering against a concurrently restarted job, so the test deliberately sets `job.retry.times = 3` and documents that one replay may consume the extra retry headroom (`mysqlcdc_to_mysql_with_sink_failure_recovery.conf:18-27`). This is a bounded race-handling policy, not an unproven guarantee.
   
   4. **F7/F8 — log rotation:** `AbstractMysqlCDCITBase.java:267-279`, especially line 275, uses `substring(Math.min(logOffset, serverLogs.length()))` inside the Awaitility lambda. If the log shrinks, the suffix becomes empty and the poll retries instead of throwing `StringIndexOutOfBoundsException`. The earlier green E2E did not force log rotation, so this is reported as defensive coverage rather than claimed path execution.
   
   Validation on the synced head: `IncrementalSourceEnumeratorTest` 2/2 passed; four affected modules Spotless and `git diff --check` passed; fork Build run 38893273138 and Apache required Build are SUCCESS; MySQL CDC E2E lanes (JDK 11 and 17) are both SUCCESS. Please re-review these pointers on the current head and refresh the recorded change request if you consider the items closed. A write-enabled maintainer approval is still required.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "li3zhi4 (via GitHub)" <gi...@apache.org>.
li3zhi4 commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-5248786758

   @DanielLeens @SEZ9 — CI on head `779c961f3` ran the full `connector-cdc-mysql-e2e` module (107 tests) and the **single failure is not from this PR**:
   
   ```
   [ERROR] MysqlCDCWithFlinkSchemaChangeIT.testMysqlCdcWithSchemaEvolutionCase:130
           ->assertSchemaEvolutionForAddColumns:256 » ConditionTimeout
   [ERROR] Tests run: 107, Failures: 0, Errors: 1, Skipped: 0
   ```
   
   - `MysqlCDCWithFlinkSchemaChangeIT` is a **pre-existing Flink-only schema-change test** (disabled for SPARK/SEATUNNEL); this PR does not touch that file at all (empty diff), and its last change predates this PR (`12ab1d077` "Stabilize flaky connector E2E tests").
   - The PR's own e2e `MysqlCDCIT#testMysqlCdcContinuesAfterSinkFailure` **passed** on the same CI run (the only error is the Flink schema-change test above).
   - The failure is a `ConditionTimeout` in `assertSchemaEvolutionForAddColumns`, consistent with the known flakiness of Flink schema-evolution e2e on loaded CI runners (the test class already carries comments about XA recover/rollback restarts on loaded runners).
   
   So the red Build is an unrelated flaky E2E, not a regression from this PR. Happy to rerun the failed jobs if the project prefers a green gate before merge; the production fix, unit test, and this PR's e2e all pass on the same run.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-5230720838

   Thanks for the thorough follow-ups — the latest commits address my concerns well, and I appreciate that the PR description now matches the traced engine behavior rather than overstating what the e2e proves.
   
   Confirming where we've landed:
   
   - **Enumerator fix**: `addSplitsBack()` being `synchronized` + re-running `assignSplits()` when `running == true` is correct defensive hardening at the contract boundary. The deterministic unit test covers the actual race independent of engine timing — that's the right primary evidence here.
   - **Reachability note**: Agreed with your re-assessment. In the clean full-pipeline restart the e2e drives (parallelism = 1), `running` is `false` at `addSplitsBack()` time and the split flows through the pre-existing `handleSplitRequest()` path; the new branch fires in partial-restart scenarios where the enumerator survives while a reader restarts. Good that this is now explicit in the description.
   - **Javadoc + `receivedReader()` comment**: The softened `run()` contract ("implementations must not rely on strict invocation order") and the cross-class invariant comment are both appropriate.
   - **e2e improvements**: Replacing the hard sleep with an await on completed checkpoint count and using the job-state transition as the failure-injection signal are solid upgrades over the previous log-keyword/sleep approach.
   
   Remaining asks before merge:
   
   1. **e2e stability**: Please run the new e2e a few times locally (or push an empty commit to re-trigger CI 2–3 times) to confirm the state-based waits don't flake around the 15s `job.retry.interval` window. Trigger-based failure injection plus timing-sensitive state assertions is the part I'd most expect to be flaky under CI load.
   2. **Test comment**: Add a short comment in the e2e test itself noting it exercises the `handleSplitRequest()` restore path and serves as a regression guard, with the unit test covering the new `running == true` branch — so future readers don't mistake it for direct coverage of the new code path.
   3. Please squash the review-fix commits into a clean history before merge.
   
   The debug-log trace of a real partial-restart interleaving remains a nice-to-have follow-up, not a blocker. Once (1) and (2) are in, this LGTM.
   
   <!-- streview-comment:103 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-5229131118

   @SEZ9 I re-checked this latest `CHANGES_REQUESTED` review against the current head (`1684c382f`, unchanged since my last pass), and it restates the exact same two findings, with the exact same wording, that you posted in your prior review yesterday (2026-08-07T10:00:15Z) — which I already traced against the actual diff and rebutted point-by-point in my review at 2026-08-07T11:15:54Z. Since the head hasn't moved, I re-verified both items again from scratch just now and the conclusion is unchanged:
   
   - **Issue 1 ("possible public API change", High/blocking):** I pulled the file straight from the PR head and the only change to `addSplitsBack()` is:
     ```java
     -    public void addSplitsBack(List<SourceSplitBase> splits, int subtaskId) {
     +    public synchronized void addSplitsBack(List<SourceSplitBase> splits, int subtaskId) {
     ```
     plus the new `if (running) { assignSplits(); }` body and an explanatory comment. Name, parameter list, and return type are byte-for-byte identical to before; `synchronized` is a method modifier, not part of the Java method signature (JLS 8.4.3.6), so it cannot break binary or source compatibility for callers or overriders. I also re-ran a repo-wide search for any class extending `IncrementalSourceEnumerator` and found zero results — there is no subclass in the tree that overrides this method, so there's no override site that could even theoretically be affected. I don't see a substantive basis for this being a blocking API break; happy to be pointed at a specific caller/override that would actually break if I'm missing something.
   - **Issue 2 ("config/option changes without doc updates", Low):** I grepped the full diff across all six changed files for any `Option`/config addition (`new Option`, `.key(`, `.defaultValue(`, etc.) — there are none. This PR touches zero configuration options anywhere, so there's nothing for docs to reflect.
   
   Since both items were already addressed with concrete evidence in the previous round and nothing in the code has changed since then, I don't think this restates a new blocker — it looks like the same review content was reposted without picking up the rebuttal. I'd suggest resolving/withdrawing the `CHANGES_REQUESTED` state unless there's a specific code path you can point to that the above doesn't cover.
   
   @li3zhi4 thanks for confirming convergence on the Issue 3 reachability note and for updating the PR description accordingly — that resolves my only remaining non-blocking concern from this side. The debug-log trace of the exact partial-restart interleaving is a nice-to-have; no need to hold this PR for it, a follow-up PR is fine.
   
   With that, my own position is unchanged from my last review: the production fix is minimal, correct, and backward-compatible, both prior test-stability issues are genuinely resolved, and I don't see any code-level blocker remaining. The only outstanding item on my side is confirming the required `Build` check is green on the current head before merge.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "li3zhi4 (via GitHub)" <gi...@apache.org>.
li3zhi4 commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-5265110113

   @SEZ9 — re: the two findings behind the standing `CHANGES_REQUESTED` (submitted against head `1684c382f`, now two commits behind), I verified both against the current head `779c961f3` and they do not hold up against the actual diff. @DanielLeens reached the same conclusion in his reviews; sharing the concrete evidence here so the review state can be reconciled.
   
   **Issue 1 (High, "Possible public API change"):** the only production-code change to `IncrementalSourceEnumerator` is `public void addSplitsBack(List<SourceSplitBase>, int)` → `public synchronized void addSplitsBack(List<SourceSplitBase>, int)` — `synchronized` is a modifier, not part of the method signature. Method name, parameter types, and return type are byte-for-byte unchanged, so no binary or source compatibility break for downstream connectors. The other production changes are comment-only: `SourceSplitEnumerator.java` (javadoc clarification, no code) and `SourceSplitEnumeratorTask.java` (7 lines of comments).
   
   **Issue 2 (Low, "Config/option changes without doc updates"):** this PR adds zero configuration surface — no `Option` class, no config key, no doc-worthy setting. The only `.conf` file in the diff is the e2e test resource `mysqlcdc_to_mysql_with_sink_failure_recovery.conf`, which is test scaffolding, not product configuration.
   
   Since both findings were repeated verbatim across the two `CHANGES_REQUESTED` reviews and the underlying concerns are resolved in the current head (which you already acknowledged in your comment — the three pre-merge asks were addressed), could you submit a fresh review on `779c961f3` or dismiss/update the existing `CHANGES_REQUESTED`? The formal review state currently blocks merge at the GitHub level (`reviewDecision: REVIEW_REQUIRED`). Happy to address anything that still looks wrong.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "li3zhi4 (via GitHub)" <gi...@apache.org>.
li3zhi4 commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-6035500141

   Mapping all three, with the honest caveat on what the single run did and didn't exercise.\n\n**1. The three robustness changes, mapped to your earlier points:**\n\n- **(a) `getServerLogs().substring(logOffset)` guard \u2014 yes, in this head.** `AbstractMysqlCDCITBase.java:275` now reads `.substring(Math.min(logOffset, serverLogs.length()))`, so a shrunk/rotated log yields \"no match yet\" and the awaitility check retries instead of `StringIndexOutOfBoundsException` aborting the wait. Being precise about your caveat: the green run did **not** exercise this path \u2014 the log didn't shrink mid-test \u2014 it's defensive hardening for a failure mode we observed as a risk, and the clamp is a one-liner to audit.\n- **(b) replay window before `DROP TRIGGER` lands \u2014 yes, handled, by the tolerant-retry option.** `mysqlcdc_to_mysql_with_sink_failure_recovery.conf` now sets `job.retry.times = 3` (lines 24-27, restored from `1` to the engine default per `EnvCommonOptions.JOB_RETRY_TIM
 ES`, `job.retry.interval.seconds = 15`), with a comment in the conf explaining the replay-burns-one-retry reasoning. So if the restarted job replays the failing insert before the `finally` block's `DROP TRIGGER` executes, that cycle fails, the job retries, and the trigger-drop (executed immediately after the checkpoint-count wait, before the next submit) makes the next cycle succeed \u2014 the test no longer fails spuriously on a capped single retry. The `DROP TRIGGER IF EXISTS` itself runs right after the 2-minute checkpoint wait (`AbstractMysqlCDCITBase.java:~234`).\n- Plus the third change: the injected-failure wait aligned from 30s to 2 minutes (`await().atMost(2, TimeUnit.MINUTES)` at `:227`).\n\n**2. `if (running)` ordering comments \u2014 already in `677216dd4f`, both sites:**\n- `run()`'s trailing `assignSplits()` (`IncrementalSourceEnumerator.java:77-83`): \"Dispatch the splits that addSplitsBack() returned before run() executed: the engine hands the checkpoint-restored spl
 its of a restarting reader to the enumerator and only invokes run() once every reader has registered, so in a restart the restored splits are normally queued in the assigner while running is still false and addSplitsBack() skipped its assignment pass. They are not lost: any split request that arrived in the meantime is already queued in readersAwaitingSplit by handleSplitRequest(), so this first assignment pass hands out the returned splits to those waiting readers.\"\n- the `if (running)` gate in `addSplitsBack()` (`:105-111`): covers both branches \u2014 `running == false` (engine delivers restored splits before `run()`; they stay queued and are dispatched by `run()`'s first pass) and `running == true` (restored split arrives after the reader's request is parked in `readersAwaitingSplit`; re-run the loop so the waiting reader gets it immediately).\n- The unit test you earlier asked for (`shouldAssignSplitsAddedBackBeforeRunExactlyOnce`) also landed before you walked it back \u2014
  it stays, it's cheap and directly pins the pre-`run()` ordering.\n\n**3. `SourceSplitEnumerator` javadoc \u2014 already narrowed in `677216dd4f` (your preferred option (a)):** the `@implNote` obligation on every implementation is gone; the contract now reads (`SourceSplitEnumerator.java:52-56`): \"The engine only guarantees that `run()` is invoked after `open()` and after every reader has been registered. It does not guarantee the order listed above: restored splits may be returned via `addSplitsBack(List, int)` at any point of the source lifecycle, for example after a checkpoint restore, before or after `run()`. How such late returned splits are dispatched to readers is left to the implementation.\" The CDC-specific stricter handling moved to `IncrementalSourceEnumerator`'s class javadoc. No PR-description statement about other connectors is needed since the API no longer promises anything beyond the engine guarantee.\n\nCI on `677216dd4f`: failures confined to the `dev`-inherited
  families (`engine-v2-it`, `all-connectors-it-2/6/7`, `paimon`, `transform-v2-it-part-1`, `unit-test windows`, plus the upstream `Dead links` badge) \u2014 `mysql-cdc-connector-it` is **not** among them. Failed-job rerun in flight; I'll post the outcome and that should be everything for your final pass.\n


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-6183297556

   @li3zhi4 Thanks for the pointer and the link to `c959fd1989`. The `substring(Math.min(logOffset, serverLogs.length()))` guard you describe is the right shape for PR11677-F7 / PR11677-F8 — a shorter log yielding an empty suffix and letting the Awaitility lambda retry is exactly what I was after. I'll confirm it against the diff on my next pass; I'm not asking for a forced-rotation test, and I appreciate you being upfront that the earlier run didn't exercise that path.
   
   Still open from the previous review:
   
   1. **PR11677-F1 / F2 (SourceSplitEnumerator javadoc):** the rewritten contract applies to every enumerator implementation, but only the CDC base enumerator was hardened. Please either narrow the javadoc to what the engine actually guarantees today (noting the CDC-side hardening as implementation behavior), or outline how the other connectors' `addSplitsBack()` implementations are expected to conform. The first option is fine for this PR.
   2. **PR11677-F3 / F4 (`if (running)` gate in `addSplitsBack()`):** splits returned before `run()` executes are dispatched only via the trailing `assignSplits()` call in `run()`. Please add a short comment documenting that invariant at the gate, and ideally a unit test covering the "addSplitsBack before run" path so it doesn't regress silently.
   3. **PR11677-F5 (30-second wait for the injected sink failure):** this is still inconsistent with the 2-minute budgets elsewhere in the same test. Either align it to the same budget or explain why 30s is sufficient here.
   4. **PR11677-F6 (replay before `DROP TRIGGER` takes effect):** a brief note on why the restart budget can't be exhausted in that window would be enough — or, if it can, a small adjustment so the failing insert can't be replayed more than once.
   
   Once those are addressed I'm happy to take another look.
   
   <!-- streview-comment:1761 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-CDC] Reassign restored splits to waiting readers after checkpoint recovery [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11677:
URL: https://github.com/apache/seatunnel/pull/11677#issuecomment-6159485179

   Thanks @li3zhi4 for the pointer list on `c959fd1989`. I haven't yet checked those ranges against the diff myself, so I'll hold off on marking anything resolved until I've read them. Concretely, I'll go through `mysqlcdc_to_mysql_with_sink_failure_recovery.conf:18-27` for the retry headroom, `SourceSplitEnumerator.java:41-58` and `SourceSplitEnumeratorTask.java:241-247` for the contract/ordering wording, `IncrementalSourceEnumerator.java:74-115` plus `IncrementalSourceEnumeratorTest.java:41-82` for the pre-run path and its tests, and `AbstractMysqlCDCITBase.java:245-289` / `:305-330` for the wait budgets.
   
   One item I don't see called out in your list: the `getServerLogs().substring(logOffset)` check in `AbstractMysqlCDCITBase.java`. On `c959fd1989`, is it guarded against the server log shrinking or rotating during the restart window so that it retries inside the awaitility wait instead of throwing `StringIndexOutOfBoundsException`? If so, could you point me to the line range?
   
   Once I've been through the ranges above and that point is clarified, I'll follow up with a final review.
   
   <!-- streview-comment:1733 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org