[jira] [Created] (FLINK-36798) Improve data processing speed during the phase from snapshot to incremental phase

["Yanquan Lv (Jira)" <ji...@apache.org>]

Yanquan Lv created FLINK-36798:
----------------------------------

             Summary: Improve data processing speed during the phase from snapshot to incremental phase
                 Key: FLINK-36798
                 URL: https://issues.apache.org/jira/browse/FLINK-36798
             Project: Flink
          Issue Type: Improvement
          Components: Flink CDC
    Affects Versions: cdc-3.1.1, cdc-3.2.0, cdc-3.1.0
            Reporter: Yanquan Lv
             Fix For: cdc-3.3.0


During the phase from snapshot to incremental phase, for each input record, we need to compare with all finished splits and find the binlog offset to check whether we should emit the record, however,  this complexity is log(n), it's a very time cost procedure.

Actually, we can improve data processing speed by the following ways:
1. For numeric fields, we can directly calculate which chunk they belong to based on the primary key and chunk size information.this complexity is O(1).
2. For non numeric fields, we can use binary search to find the shard to which the data belongs. this complexity is log(n).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)