[jira] [Created] (FLINK-36682) Add split assign strategy to avoid OOM error in TaskManager

["Yanquan Lv (Jira)" <ji...@apache.org>]

Yanquan Lv created FLINK-36682:
----------------------------------

             Summary: Add split assign strategy to avoid OOM error in TaskManager
                 Key: FLINK-36682
                 URL: https://issues.apache.org/jira/browse/FLINK-36682
             Project: Flink
          Issue Type: Bug
          Components: Flink CDC
    Affects Versions: cdc-3.3.0
            Reporter: Yanquan Lv
             Fix For: cdc-3.3.0


During snapshot reading phase, we will split table into chunks and assign them to split reader in TaskManager.

For evenly chunk split, them are assigned in ascending order. For example, a table that primary key is id may be split into chunks like [-∞, 10000), [10000,20000), [20000,30000), ......[1500000, +∞). However, during snapshot reading phase, more records may be inserted and id will increase to relative high, and the last split may need to fetch too many records, for example, the last split may need to fetch records in range [1500000, 3000000], witch will cause TaskManager out of memory.

So I propose to add a strategy to allow user to config how to assign split, and by default, we can send the last split first to split reader.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)