[jira] [Updated] (FLINK-40741) [Flink cdc]After the full snapshot read of the newly added table completes, the newly added table is no longer synchronized, while the existing (old) tables continue reading

["Ruan Hang (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-40741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ruan Hang updated FLINK-40741:
------------------------------
    Description: 
The table A had its schema modified last night, after which the original job started throwing errors. When this table was newly added to the CDC job, no newly-added (incremental) data was being synchronized for it — the full snapshot data appeared to sync normally. Looking at the TaskManager logs, all other tables were syncing normally, but this table indeed had no sink synchronization, leading to the suspicion that even the full snapshot data had not been synchronized.

   Root Cause:

   1. When the user reads the full snapshot data of a newly added table, each snapshot split reports back once it finishes reading.
   2. Once all snapshot splits have finished reading, the enumerator transitions into the NEWLY_ADDED_ASSIGNING_SNAPSHOT_FINISHED state.
   3. The enumerator then periodically checks whether it is in the NEWLY_ADDED_ASSIGNING_SNAPSHOT_FINISHED state. If it is:
     - a. It sends a BinlogSplitUpdateRequest to make the binlog reader pause reading.
     - b. Through a LatestFinishedSplitsNumberEvent, it tells the binlog reader that there is a newly added table and updates the latest finished-splits number for that table; only after this can the newly added
   table's LatestFinishedSplitsNumber be read.
     - d. It transitions into NEWLY_ADDED_ASSIGNING_FINISHED, indicating that the incremental phase is complete.
     - e. The reader then re-reads the binlog of the new table.

   However, the job experienced a failover at step 4, when it transitioned into the NEWLY_ADDED_ASSIGNING_FINISHED state. The job's strategy is region failover, so the enumerator is not restarted; instead,
   SplitEnumerator#addSplitsBack is called. At this point the state remains NEWLY_ADDED_ASSIGNING_FINISHED, while the reader, after restarting, rolls back to the state at step 2.

   Then, at step 3, the check finds that the state is not NEWLY_ADDED_ASSIGNING_SNAPSHOT_FINISHED, so the binlog is not updated. As a result, the incremental data of the new table is never read.

> [Flink cdc]After the full snapshot read of the newly added table completes, the newly added table is no longer synchronized, while the existing (old) tables continue reading
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-40741
>                 URL: https://issues.apache.org/jira/browse/FLINK-40741
>             Project: Flink
>          Issue Type: Bug
>          Components: Flink CDC
>            Reporter: Ruan Hang
>            Priority: Major
>
> The table A had its schema modified last night, after which the original job started throwing errors. When this table was newly added to the CDC job, no newly-added (incremental) data was being synchronized for it — the full snapshot data appeared to sync normally. Looking at the TaskManager logs, all other tables were syncing normally, but this table indeed had no sink synchronization, leading to the suspicion that even the full snapshot data had not been synchronized.
>    Root Cause:
>    1. When the user reads the full snapshot data of a newly added table, each snapshot split reports back once it finishes reading.
>    2. Once all snapshot splits have finished reading, the enumerator transitions into the NEWLY_ADDED_ASSIGNING_SNAPSHOT_FINISHED state.
>    3. The enumerator then periodically checks whether it is in the NEWLY_ADDED_ASSIGNING_SNAPSHOT_FINISHED state. If it is:
>      - a. It sends a BinlogSplitUpdateRequest to make the binlog reader pause reading.
>      - b. Through a LatestFinishedSplitsNumberEvent, it tells the binlog reader that there is a newly added table and updates the latest finished-splits number for that table; only after this can the newly added
>    table's LatestFinishedSplitsNumber be read.
>      - d. It transitions into NEWLY_ADDED_ASSIGNING_FINISHED, indicating that the incremental phase is complete.
>      - e. The reader then re-reads the binlog of the new table.
>    However, the job experienced a failover at step 4, when it transitioned into the NEWLY_ADDED_ASSIGNING_FINISHED state. The job's strategy is region failover, so the enumerator is not restarted; instead,
>    SplitEnumerator#addSplitsBack is called. At this point the state remains NEWLY_ADDED_ASSIGNING_FINISHED, while the reader, after restarting, rolls back to the state at step 2.
>    Then, at step 3, the check finds that the state is not NEWLY_ADDED_ASSIGNING_SNAPSHOT_FINISHED, so the binlog is not updated. As a result, the incremental data of the new table is never read.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)