You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "Zhongmin Qiao (Jira)" <ji...@apache.org> on 2024/07/22 13:14:00 UTC
[jira] [Created] (FLINK-35874) Check pureBinlogPhaseTables set before call getBinlogPosition method in BinlogSplitReader
Zhongmin Qiao created FLINK-35874:
-------------------------------------
Summary: Check pureBinlogPhaseTables set before call getBinlogPosition method in BinlogSplitReader
Key: FLINK-35874
URL: https://issues.apache.org/jira/browse/FLINK-35874
Project: Flink
Issue Type: Improvement
Components: Flink CDC
Reporter: Zhongmin Qiao
Attachments: image-2024-07-22-19-26-59-158.png, image-2024-07-22-19-27-19-366.png, image-2024-07-22-19-30-08-989.png, image-2024-07-22-19-36-20-481.png, image-2024-07-22-19-36-40-581.png, image-2024-07-22-19-37-35-542.png, image-2024-07-22-21-12-03-316.png
The method getBinlogPosition of RecordUtil which is called by BinlogSplitReader.
shouldEmit is a highly performance-consuming method. This is because it iterates through the sourceOffset map of the SourceRecord, and during the iteration, it also performs a toString() conversion on the value. Finally, it calls the putAll method of BinlogOffsetBuilder to put all the elements obtained from the iteration into the offsetMap (which involves another map traversal and hashcode computation). Despite the significant performance impact of getBinlogPosition, we still need to call it when emitting each DataChangeRecord, which reduces the efficiency of data processing in Flink CDC.
!image-2024-07-22-19-26-59-158.png|width=545,height=222!
!image-2024-07-22-19-27-19-366.png|width=545,height=119!
However, we can optimize and avoid frequent invocations of getBinlogPosition by moving the check pureBinlogPhaseTables.contains(tableId) in the hasEnterPureBinlogPhase method before calling getBinlogPosition. This way, if the SourceRecord belongs to a pure binlog phase table, we can directly return true without the need for the highly performance-consuming getBinlogPosition method.
diff
!image-2024-07-22-21-12-03-316.png|width=548,height=236!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)