You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "wuhainan (via GitHub)" <gi...@apache.org> on 2026/06/05 05:06:34 UTC

[I] [Bug] [Connector-V2] [TiDB-CDC] resolvedTs keeps advancing but row change events are silently missed [seatunnel]

wuhainan opened a new issue, #11013:
URL: https://github.com/apache/seatunnel/issues/11013

   ### Search before asking
   
   - [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
   
   
   ### What happened
   
   A TiDB-CDC job stayed RUNNING and checkpoints kept succeeding, but the target table stopped receiving new row changes.
   
   The source table continued to receive inserts/updates after the cutoff time, while the target table remained stale.
   
   In this case:
   
   - Target table stopped at `max(updated_at) = 2026-06-03 18:59:55`.
   - Source table continued changing until at least `max(updated_at) = 2026-06-05 11:27:36`.
   - There were 1183 source rows changed after the cutoff time.
   - The Flink job remained RUNNING.
   - Checkpoints kept completing successfully.
   - No sink errors or backpressure were observed.
   - Flink metrics stopped increasing:
     - Source `numRecordsOut = 91746`
     - Sink `numRecordsIn = 91746`
     - SourceReceivedQPS = 0.0
     - SinkWriteQPS = 0.0
     - Sink error count = 0
   
   TaskManager logs show that `TiDBSourceReader` and `CDCClient` continued advancing `resolvedTs` after the target stopped receiving rows:
   
   TiDBSourceReader - Capture streaming event from resolvedTs:...
   CDCClient - handle resolvedTs: ..., regionId: ...
   TiDBSourceReader - Capture streaming event next resolvedTs:...
   
   ### SeaTunnel Version
   
   2.3.13
   
   ### SeaTunnel Config
   
   ```conf
   env {
     parallelism = 1
     job.mode = "STREAMING"
     job.name = "dsp-17-v2-wzb_test_deposit_applications"
     checkpoint.interval = 300000
     checkpoint.timeout = 600000
     checkpoint.mode = "EXACTLY_ONCE"
     restart-strategy = "failure-rate"
     restart-strategy.failure-rate.max-failures-per-interval = 10
     restart-strategy.failure-rate.failure-rate-interval = "300 s"
     restart-strategy.failure-rate.delay = "10 s"
   }
   
   source {
     TiDB-CDC {
       plugin_output = "src"
       url = "jdbc:mysql://10.19.20.11:4001/alpha_online"
       driver = "com.mysql.cj.jdbc.Driver"
       pd-addresses = "10.19.20.23:2379,10.19.20.61:2379,10.19.20.152:2379"
       username = "******"
       password = "******"
       database-name = "alpha_online"
       table-name = "deposit_applications"
       startup.mode = "initial"
     }
   }
   
   sink {
     Jdbc {
       source_table_name = "src"
       driver = "com.mysql.cj.jdbc.Driver"
       url = "jdbc:mysql://10.19.20.7:4000/?useSSL=false&useUnicode=true&characterEncoding=utf8"
       user = "root"
       password = "******"
       batch_size = 1000
       database = "sync_test"
       table = "deposit_applications"
       primary_keys = ["id"]
       generate_sink_sql = true
       support_upsert_by_query_primary_key_exist = true
     }
   }
   ```
   
   ### Running Command
   
   ```shell
   The job was submitted by our internal DataT platform to Flink on YARN.
   
   Runtime application:
   application_1756448821197_5312
   
   Flink job name:
   dsp-17-v2-wzb_test_deposit_applications
   
   The platform uses SeaTunnel Flink starter with the TiDB-CDC source plugin.
   The exact generated SeaTunnel config is provided in the "SeaTunnel Config" section above.
   
   This was not started by manually running start-seatunnel-flink-*.sh.
   ```
   
   ### Error Exception
   
   ```log
   No exception was thrown.
   
   The job stayed RUNNING and checkpoints kept succeeding.
   No sink errors or backpressure were observed.
   
   The issue is silent data loss / silent data miss:
   TiDBSourceReader and CDCClient kept advancing resolvedTs, but row change events after 2026-06-03 18:59:55 were not emitted downstream.
   ```
   
   ### Zeta or Flink or Spark Version
   
   Flink 1.17.1
   
   ### Java or Scala Version
   
   Java 8,OpenJDK 1.8.0_422
   
   ### Screenshots
   
   [jobmanager-log-excerpt.log](https://github.com/user-attachments/files/28624318/jobmanager-log-excerpt.log)
   
   [taskmanager-log-excerpt.log](https://github.com/user-attachments/files/28624325/taskmanager-log-excerpt.log)
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [Connector-V2] [TiDB-CDC] resolvedTs keeps advancing but row change events are silently missed [seatunnel]

Posted by "Huaian666 (via GitHub)" <gi...@apache.org>.
Huaian666 commented on issue #11013:
URL: https://github.com/apache/seatunnel/issues/11013#issuecomment-4637318188

   performance surprised me tbh. expected it to be slower than what we were using but it's noticeably faster in our benchmarks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [Connector-V2] [TiDB-CDC] resolvedTs keeps advancing but row change events are silently missed [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #11013:
URL: https://github.com/apache/seatunnel/issues/11013#issuecomment-4696883287

   Thanks — this is enough useful evidence for the issue to stay actionable.
   
   With the newer single-split reproduction, the problem is no longer just "the job kept running" in a broad sense. The combination of:
   - one split
   - `startup.mode = initial`
   - snapshot start being logged
   - downstream rows stalling far below the source count
   - `resolvedTs` and checkpoints continuing to advance
   
   still points much more strongly to a real reader-side bug in the snapshot-to-stream handoff / row-flush path than to a monitoring illusion.
   
   So from my side, no additional generic information is required right now just to keep this issue open. The logs and count SQL you preserved are already valuable enough for someone preparing a deterministic regression test or a fix PR.
   
   If a contributor starts working on the fix later, the full JobManager / TaskManager logs around this reproduction would still be helpful as follow-up evidence. But this issue already has enough signal to continue tracking as a real bug.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [Connector-V2] [TiDB-CDC] resolvedTs keeps advancing but row change events are silently missed [seatunnel]

Posted by "wuhainan (via GitHub)" <gi...@apache.org>.
wuhainan commented on issue #11013:
URL: https://github.com/apache/seatunnel/issues/11013#issuecomment-4641280555

   > That additional observation is useful.
   > 
   > I checked the current reader flow more closely, and I do not see an intentional handoff that marks the snapshot as finished early in the normal path. In `INITIAL` mode, the reader only flips `snapshotCompleted` after `snapshotEvents(...)` returns, and `resolvedTs` is only advanced after the snapshot scan reaches the end of the current split.
   > 
   > So the `~91,746` ceiling you observed is more likely a symptom of the same underlying reader bug family rather than a separate "expected switch to streaming" path. In other words, this still looks like either:
   > 
   > 1. the snapshot phase exited prematurely for some split, or
   > 2. the reader reached streaming with an already incomplete materialization state and then continued advancing `resolvedTs`.
   > 
   > The most useful next evidence, if you still have it, would be:
   > 
   > 1. the startup log lines around `[splitId] Snapshot events start.` and the first `Capture streaming event from resolvedTs` for the same split
   > 2. how many source splits were assigned for that table in the job
   > 
   > That will let us distinguish "snapshot never fully drained" from "streaming advanced after a partial snapshot" much more cleanly. Either way, this still looks like a real TiDB-CDC reader bug worth keeping open.
   
   I started a new isolated reproduction job and collected cleaner evidence.
   
   Job:
   
   - Application: `application_1756448821197_5356`
   - Job name: `tidb2tidb_deposit_applications`
   - SeaTunnel connector: `connector-cdc-tidb-2.3.13`
   - TiKV client: `tikv-client-java-3.2.0`
   - `startup.mode = initial`
   - Flink source parallelism: `1`
   
   The source table row count was:
   
       alpha_online.deposit_applications: 522,625 rows
   
   The target table stopped at:
   
       target table: 93,354 rows
   
   The Flink metrics also stopped at exactly the same number:
   
       Source numRecordsOut = 93354
       Sink numRecordsIn = 93354
       SourceReceivedQPS = 0.0
       SinkWriteQPS = 0.0
       Sink numRecordsSendErrors = 0
       backPressuredTimeMsPerSecond = 0
       idleTimeMsPerSecond = 1000
   
   I queried the same metrics twice and the values stayed at `93354`, so the job was no longer emitting rows downstream.
   
   Startup log evidence:
   
   2026-06-07 10:46:40,705 INFO  org.apache.seatunnel.connectors.seatunnel.cdc.tidb.source.reader.TiDBSourceReader [] - [alpha_online:deposit_applications:<ByteString@446b0db2 size=19 contents="t\200\000\000\000\000\000e\377_r\000\000\000\000\000\000\000\000">-<ByteString@2828c34c size=19 contents="t\200\000\000\000\000\000e\377_s\000\000\000\000\000\000\000\000">] Snapshot events start.
   2026-06-07 10:46:41,460 WARN  org.apache.seatunnel.shade.com.zaxxer.hikari.HikariConfig    [] - HikariPool-1 - idleTimeout has been set but has no effect because the pool is operating as a fixed size pool.
   2026-06-07 10:46:41,460 INFO  org.apache.seatunnel.shade.com.zaxxer.hikari.HikariDataSource [] - HikariPool-1 - Starting...
   2026-06-07 10:46:41,474 INFO  org.apache.seatunnel.shade.com.zaxxer.hikari.HikariDataSource [] - HikariPool-1 - Start completed.
   2026-06-07 10:46:59,162 INFO  org.apache.seatunnel.connectors.seatunnel.cdc.tidb.source.reader.TiDBSourceReader [] - Capture streaming event from resolvedTs:466826140257353737
   2026-06-07 10:46:59,171 INFO  org.tikv.cdc.CDCClient                                       [] - remove regions: []
   2026-06-07 10:46:59,171 INFO  org.tikv.cdc.CDCClient                                       [] - add regions: [{Region[1485688] ConfVer[313901] Version[12789] Store[177965148] KeyRange[t\200\000\000\000\000\000e\377_r\200\000\000\000\000\003Z\247]:[t\200\000\000\000\000\000e\377_r\200\000\000\000\000\f\206\233]}, {Region[1749236] ConfVer[313949] Version[12781] Store[177965148] KeyRange[t\200\000\000\000\000\000e\377_r\200\000\000\000\00029\351]:[t\200\000\000\000\000\000f\001_i\200\000\000\000\000\000\000\002\000\003\200\000\000\000\000\000C\225]}, {Region[164304772] ConfVer[313907] Version[12784] Store[177965147] KeyRange[t\200\000\000\000\000\000e\377_i\200\000\000\000\000\000\000\004\004\000\000\027\365|\305\270\375\003\200\000\000\000\000\037!\316]:[t\200\000\000\000\000\000e\377_r\200\000\000\000\000\003Z\247]}, {Region[164304978] ConfVer[313919] Version[12781] Store[164454270] KeyRange[t\200\000\000\000\000\000e\377_r\200\000\000\000\000\f\206\233]:[t\200\000\000\000\000\000e
 \377_r\200\000\000\000\000\025\351\000]}, {Region[575192972] ConfVer[313955] Version[12781] Store[177965147] KeyRange[t\200\000\000\000\000\000e\377_r\200\000\000\000\000\025\351\000]:[t\200\000\000\000\000\000e\377_r\200\000\000\000\00029\351]}], timestamp: 466826140257353737
   2026-06-07 10:46:59,175 INFO  org.tikv.cdc.RegionCDCClient                                 [] - start streaming region: 1485688, running: true
   2026-06-07 10:46:59,181 INFO  org.tikv.cdc.RegionCDCClient                                 [] - start streaming region: 1749236, running: true
   2026-06-07 10:46:59,181 INFO  org.tikv.cdc.RegionCDCClient                                 [] - start streaming region: 164304772, running: true
   2026-06-07 10:46:59,181 INFO  org.tikv.cdc.RegionCDCClient                                 [] - start streaming region: 164304978, running: true
   2026-06-07 10:46:59,181 INFO  org.tikv.cdc.RegionCDCClient                                 [] - start streaming region: 575192972, running: true
   2026-06-07 10:46:59,182 INFO  org.tikv.cdc.CDCClient                                       [] - keyRange applied
   2026-06-07 10:46:59,282 INFO  org.apache.seatunnel.connectors.seatunnel.cdc.tidb.source.reader.TiDBSourceReader [] - Capture streaming event next resolvedTs:466826140257353737
   2026-06-07 10:46:59,284 INFO  org.apache.seatunnel.connectors.seatunnel.cdc.tidb.source.reader.TiDBSourceReader [] - Capture streaming event from resolvedTs:466826140257353737
   2026-06-07 10:46:59,384 INFO  org.apache.seatunnel.connectors.seatunnel.cdc.tidb.source.reader.TiDBSourceReader [] - Capture streaming event next resolvedTs:466826140257353737
   2026-06-07 10:47:00,385 INFO  org.apache.seatunnel.connectors.seatunnel.cdc.tidb.source.reader.TiDBSourceReader [] - Capture streaming event from resolvedTs:466826140257353737
   2026-06-07 10:47:00,386 INFO  org.tikv.cdc.CDCClient                                       [] - handle resolvedTs: 466826145175175177, regionId: 575192972
   
   So the reader entered streaming about 18 seconds after `Snapshot events start`, while only about `93,354 / 522,625` rows were materialized downstream.
   
   For the source split count:
   
   - The Flink execution plan shows source parallelism was `1`.
   - The TaskManager log contains only one occurrence of `Snapshot events start`.
   - The 5 regions shown in the CDCClient log are TiKV regions watched by CDCClient for that one split, not SeaTunnel source splits.
   
   After that, `resolvedTs` kept advancing:
   
       2026-06-07 10:56:55,574 INFO  TiDBSourceReader - Capture streaming event next resolvedTs:466826177279950865
       2026-06-07 10:56:57,574 INFO  TiDBSourceReader - Capture streaming event next resolvedTs:466826177355972613
   
   Checkpoints also kept succeeding:
   
       2026-06-07 10:56:29,604 INFO  CheckpointCoordinator - Completed checkpoint 20
       2026-06-07 10:56:59,591 INFO  CheckpointCoordinator - Completed checkpoint 21
   
   This looks like a reproducible case where the reader entered the streaming phase before the initial snapshot was fully materialized downstream.
   
   I still cannot prove the exact internal reason from logs alone because there is no explicit `Snapshot events finished` log or snapshot row count, but the evidence strongly suggests that the snapshot phase exited prematurely for the split, then the reader continued advancing `resolvedTs` in streaming mode.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [Connector-V2] [TiDB-CDC] resolvedTs keeps advancing but row change events are silently missed [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo closed issue #11013: [Bug] [Connector-V2] [TiDB-CDC] resolvedTs keeps advancing but row change events are silently missed
URL: https://github.com/apache/seatunnel/issues/11013


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [Connector-V2] [TiDB-CDC] resolvedTs keeps advancing but row change events are silently missed [seatunnel]

Posted by "wuhainan (via GitHub)" <gi...@apache.org>.
wuhainan commented on issue #11013:
URL: https://github.com/apache/seatunnel/issues/11013#issuecomment-4645128312

   > Thanks for collecting the cleaner reproduction data — this is much stronger evidence.
   > 
   > The new logs make the direction here considerably narrower. In the current reader path, entering streaming mode is not by itself proof that the initial snapshot was fully materialized downstream. `resolvedTs` can continue to move forward once the CDC client starts, while actual row emission still depends on the PREWRITE/COMMIT matching and flush path. That means the symptom you captured is consistent with a real reader-side bug, not just a metrics illusion.
   > 
   > The most important part of your new report is this combination:
   > 
   > * one source split
   > * `startup.mode = initial`
   > * snapshot starts
   > * streaming begins about 18 seconds later
   > * downstream row count stalls at `93,354` while the source table had `522,625` rows
   > * `resolvedTs` and checkpoints keep advancing afterward
   > 
   > That strongly suggests the problem is in the snapshot-to-stream handoff family, rather than a normal or expected transition. In other words, this now looks much closer to either:
   > 
   > 1. the snapshot phase exiting before the split was fully materialized, or
   > 2. the reader entering streaming with incomplete materialization state and then continuing to advance `resolvedTs`.
   > 
   > So this issue is worth keeping separate and open. It is related to `#8815`, but the new evidence here is much more specific and should help us drive a more targeted fix.
   > 
   > The next high-value step would be a deterministic regression test around the single-split `INITIAL` path, especially validating that the reader does not enter effective streaming progress before the snapshot for that split is fully drained downstream.
   
   @DanielLeens Thank you for the detailed analysis and confirmation.
   
   I will keep the reproduction job logs and metrics available. If a deterministic regression test or a fix PR needs more runtime evidence, I can provide the full JobManager / TaskManager logs and the source/target count SQL results.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [Connector-V2] [TiDB-CDC] resolvedTs keeps advancing but row change events are silently missed [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #11013:
URL: https://github.com/apache/seatunnel/issues/11013#issuecomment-4636756956

   I traced the current TiDB CDC reader path, and this does look like a real source-side bug worth tracking, not just a sink-side stall.
   
   In the current implementation, `TiDBSourceReader.captureStreamingEvents(...)` always advances the split's `resolvedTs` from `cdcClient.getMaxResolvedTs()`, but actual downstream emission still depends on the `preWrites` / `commits` matching path and `flushRows(resolvedTs)`. That means the symptom you reported is plausible in the current code: checkpoints can keep succeeding and `resolvedTs` can keep moving forward even while some row changes are no longer materialized downstream.
   
   This also looks related to #8815, but not identical. #8815 was centered on older TiDB-CDC-MIGRATE loss reports, while your reproduction is on 2.3.13 + Flink + normal TiDB-CDC and already narrows the symptom much better.
   
   We should keep this open as a real bug. We have labeled it as `help wanted` since the reporter did not opt into a PR. A good next debugging slice would be to verify whether the loss happens around the `PREWRITE` / `COMMIT` matching path during region movement or client pull gaps, because that is the part that can let `resolvedTs` advance independently from emitted rows.
   
   If someone wants to work on a fix, a small first PR with a deterministic regression test around this reader path would be very valuable.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [Connector-V2] [TiDB-CDC] resolvedTs keeps advancing but row change events are silently missed [seatunnel]

Posted by "wuhainan (via GitHub)" <gi...@apache.org>.
wuhainan commented on issue #11013:
URL: https://github.com/apache/seatunnel/issues/11013#issuecomment-4637008473

   > I traced the current TiDB CDC reader path, and this does look like a real source-side bug worth tracking, not just a sink-side stall.
   > 
   > In the current implementation, `TiDBSourceReader.captureStreamingEvents(...)` always advances the split's `resolvedTs` from `cdcClient.getMaxResolvedTs()`, but actual downstream emission still depends on the `preWrites` / `commits` matching path and `flushRows(resolvedTs)`. That means the symptom you reported is plausible in the current code: checkpoints can keep succeeding and `resolvedTs` can keep moving forward even while some row changes are no longer materialized downstream.
   > 
   > This also looks related to [#8815](https://github.com/apache/seatunnel/issues/8815), but not identical. [#8815](https://github.com/apache/seatunnel/issues/8815) was centered on older TiDB-CDC-MIGRATE loss reports, while your reproduction is on 2.3.13 + Flink + normal TiDB-CDC and already narrows the symptom much better.
   > 
   > We should keep this open as a real bug. We have labeled it as `help wanted` since the reporter did not opt into a PR. A good next debugging slice would be to verify whether the loss happens around the `PREWRITE` / `COMMIT` matching path during region movement or client pull gaps, because that is the part that can let `resolvedTs` advance independently from emitted rows.
   > 
   > If someone wants to work on a fix, a small first PR with a deterministic regression test around this reader path would be very valuable.
   
   Thanks for the confirmation.
   
   I would like to add one more observation: this job was configured with `startup.mode = "initial"`, but it looks like the reader switched to streaming before the initial snapshot was fully completed.
   
   The source table had about 521,960 rows, while the target table only had about 91,746 rows when downstream emission stopped. The Flink source/sink metrics were also around 91,746 records.
   
   So it seems that only part of the initial snapshot was emitted before the reader entered the incremental CDC phase. I am not sure whether this is the same root cause as the `resolvedTs` advancing issue, but it may be related.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [Connector-V2] [TiDB-CDC] resolvedTs keeps advancing but row change events are silently missed [seatunnel]

Posted by "Huaian666 (via GitHub)" <gi...@apache.org>.
Huaian666 commented on issue #11013:
URL: https://github.com/apache/seatunnel/issues/11013#issuecomment-4637771034

   performance surprised me tbh. expected it to be slower than what we were using but it's noticeably faster in our benchmarks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [Connector-V2] [TiDB-CDC] resolvedTs keeps advancing but row change events are silently missed [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #11013:
URL: https://github.com/apache/seatunnel/issues/11013#issuecomment-4640972087

   That additional observation is useful.
   
   I checked the current reader flow more closely, and I do not see an intentional handoff that marks the snapshot as finished early in the normal path. In `INITIAL` mode, the reader only flips `snapshotCompleted` after `snapshotEvents(...)` returns, and `resolvedTs` is only advanced after the snapshot scan reaches the end of the current split.
   
   So the `~91,746` ceiling you observed is more likely a symptom of the same underlying reader bug family rather than a separate "expected switch to streaming" path. In other words, this still looks like either:
   1. the snapshot phase exited prematurely for some split, or
   2. the reader reached streaming with an already incomplete materialization state and then continued advancing `resolvedTs`.
   
   The most useful next evidence, if you still have it, would be:
   1. the startup log lines around `[splitId] Snapshot events start.` and the first `Capture streaming event from resolvedTs` for the same split
   2. how many source splits were assigned for that table in the job
   
   That will let us distinguish "snapshot never fully drained" from "streaming advanced after a partial snapshot" much more cleanly. Either way, this still looks like a real TiDB-CDC reader bug worth keeping open.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [Connector-V2] [TiDB-CDC] resolvedTs keeps advancing but row change events are silently missed [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #11013:
URL: https://github.com/apache/seatunnel/issues/11013#issuecomment-4644692689

   Thanks for collecting the cleaner reproduction data — this is much stronger evidence.
   
   The new logs make the direction here considerably narrower. In the current reader path, entering streaming mode is not by itself proof that the initial snapshot was fully materialized downstream. `resolvedTs` can continue to move forward once the CDC client starts, while actual row emission still depends on the PREWRITE/COMMIT matching and flush path. That means the symptom you captured is consistent with a real reader-side bug, not just a metrics illusion.
   
   The most important part of your new report is this combination:
   - one source split
   - `startup.mode = initial`
   - snapshot starts
   - streaming begins about 18 seconds later
   - downstream row count stalls at `93,354` while the source table had `522,625` rows
   - `resolvedTs` and checkpoints keep advancing afterward
   
   That strongly suggests the problem is in the snapshot-to-stream handoff family, rather than a normal or expected transition. In other words, this now looks much closer to either:
   1. the snapshot phase exiting before the split was fully materialized, or
   2. the reader entering streaming with incomplete materialization state and then continuing to advance `resolvedTs`.
   
   So this issue is worth keeping separate and open. It is related to `#8815`, but the new evidence here is much more specific and should help us drive a more targeted fix.
   
   The next high-value step would be a deterministic regression test around the single-split `INITIAL` path, especially validating that the reader does not enter effective streaming progress before the snapshot for that split is fully drained downstream.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org