[jira] [Updated] (FLINK-39207) MySql cdc connector could get stuck in backfill binlog reading after a failover within snapshot phase

["Cong Cheng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-39207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Cong Cheng updated FLINK-39207:
-------------------------------
    Description: 
h2. Summary

When a `MySqlSourceReader` processes multiple snapshot splits sequentially using the same `SnapshotSplitReader` instance (typically after a failover in snapshot phase), the reader gets stuck and hangs indefinitely (stops emitting records).
h2. Root Cause Analysis
 # When the snapshot split is large or with poor network conditions, a snapshot split will take more time to finished, while processing the current snapshot split, a task failover (caused by machine down, lost connection to server etc.) could happen. The task will try to recovery with the unfinished snapshot split；
 # After the recovery, the `MySqlSourceReader` will send another split request since current assigned split number is less than 1 (the behaviour is introduced by [this pull request|https://github.com/apache/flink-cdc/pull/1927] ), thus the MySqlSourceReader could be recovered with two snapshot splits;
 # When the first split finishes, `stopCurrentTask()` is called, which invokes `changeEventSourceContext.stopChangeEventSource()` . This sets the internal isRunning flag of the context to false;
 # When `submitSplit()` is called for the next split, it reuses the same SnapshotSplitReader instance and the same changeEventSourceContext. However, the submitSplit() method fails to reset the isRunning flag of the context back to true;
 # When the backfill task (which is MySqlBinlogSplitReadTask ) for the new snapshot split starts, it checks context.isRunning() , finds it to be false , and exits immediately without reading any data or sending the BINLOG_END event;
 # The main thread ( pollWithBuffer ) waits indefinitely for data or the end signal from the queue, leading to a deadlock.

h2. Steps to Reproduce
 # Configure a MySQL CDC Source with scan.incremental.snapshot.chunk.size set to a large value to ensure a snapshot split is time consuming to read;
 # Trigger a TaskManager failover while the job is in the snapshot phase;
 # Observe that the job hangs after processing the first split.

 

 

  was:
h2. Summary

When a `MySqlSourceReader` processes multiple snapshot splits sequentially using the same `SnapshotSplitReader` instance (typically after a failover in snapshot phase), the reader gets stuck and hangs indefinitely (stops emitting records).
h2. Problem Statement


> MySql cdc connector could get stuck in backfill binlog reading after a failover within snapshot phase
> -----------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-39207
>                 URL: https://issues.apache.org/jira/browse/FLINK-39207
>             Project: Flink
>          Issue Type: Bug
>          Components: Flink CDC
>    Affects Versions: cdc-3.1.0, cdc-3.2.0, cdc-3.1.1, cdc-3.3.0, cdc-3.2.1, cdc-3.4.0, cdc-3.5.0
>            Reporter: Cong Cheng
>            Priority: Major
>
> h2. Summary
> When a `MySqlSourceReader` processes multiple snapshot splits sequentially using the same `SnapshotSplitReader` instance (typically after a failover in snapshot phase), the reader gets stuck and hangs indefinitely (stops emitting records).
> h2. Root Cause Analysis
>  # When the snapshot split is large or with poor network conditions, a snapshot split will take more time to finished, while processing the current snapshot split, a task failover (caused by machine down, lost connection to server etc.) could happen. The task will try to recovery with the unfinished snapshot split；
>  # After the recovery, the `MySqlSourceReader` will send another split request since current assigned split number is less than 1 (the behaviour is introduced by [this pull request|https://github.com/apache/flink-cdc/pull/1927] ), thus the MySqlSourceReader could be recovered with two snapshot splits;
>  # When the first split finishes, `stopCurrentTask()` is called, which invokes `changeEventSourceContext.stopChangeEventSource()` . This sets the internal isRunning flag of the context to false;
>  # When `submitSplit()` is called for the next split, it reuses the same SnapshotSplitReader instance and the same changeEventSourceContext. However, the submitSplit() method fails to reset the isRunning flag of the context back to true;
>  # When the backfill task (which is MySqlBinlogSplitReadTask ) for the new snapshot split starts, it checks context.isRunning() , finds it to be false , and exits immediately without reading any data or sending the BINLOG_END event;
>  # The main thread ( pollWithBuffer ) waits indefinitely for data or the end signal from the queue, leading to a deadlock.
> h2. Steps to Reproduce
>  # Configure a MySQL CDC Source with scan.incremental.snapshot.chunk.size set to a large value to ensure a snapshot split is time consuming to read;
>  # Trigger a TaskManager failover while the job is in the snapshot phase;
>  # Observe that the job hangs after processing the first split.
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)