You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "Zhongmin Qiao (Jira)" <ji...@apache.org> on 2024/07/22 13:14:00 UTC

[jira] [Created] (FLINK-35874) Check pureBinlogPhaseTables set before call getBinlogPosition method in BinlogSplitReader

Zhongmin Qiao created FLINK-35874:
-------------------------------------

             Summary: Check pureBinlogPhaseTables set before call getBinlogPosition method in BinlogSplitReader
                 Key: FLINK-35874
                 URL: https://issues.apache.org/jira/browse/FLINK-35874
             Project: Flink
          Issue Type: Improvement
          Components: Flink CDC
            Reporter: Zhongmin Qiao
         Attachments: image-2024-07-22-19-26-59-158.png, image-2024-07-22-19-27-19-366.png, image-2024-07-22-19-30-08-989.png, image-2024-07-22-19-36-20-481.png, image-2024-07-22-19-36-40-581.png, image-2024-07-22-19-37-35-542.png, image-2024-07-22-21-12-03-316.png

The method getBinlogPosition of RecordUtil which is called by  BinlogSplitReader.
shouldEmit is a highly performance-consuming method. This is because it iterates through the sourceOffset map of the SourceRecord, and during the iteration, it also performs a toString() conversion on the value. Finally, it calls the putAll method of BinlogOffsetBuilder to put all the elements obtained from the iteration into the offsetMap (which involves another map traversal and hashcode computation). Despite the significant performance impact of getBinlogPosition, we still need to call it when emitting each DataChangeRecord, which reduces the efficiency of data processing in Flink CDC.
!image-2024-07-22-19-26-59-158.png|width=545,height=222!

!image-2024-07-22-19-27-19-366.png|width=545,height=119!

However, we can optimize and avoid frequent invocations of getBinlogPosition by moving the check pureBinlogPhaseTables.contains(tableId) in the hasEnterPureBinlogPhase method before calling getBinlogPosition. This way, if the SourceRecord belongs to a pure binlog phase table, we can directly return true without the need for the highly performance-consuming getBinlogPosition method.

diff

!image-2024-07-22-21-12-03-316.png|width=548,height=236!

 

 

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)