You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Zhongmin Qiao (Jira)" <ji...@apache.org> on 2024/07/23 02:22:00 UTC
[jira] [Commented] (FLINK-35874) Check pureBinlogPhaseTables set before call getBinlogPosition method in BinlogSplitReader
[ https://issues.apache.org/jira/browse/FLINK-35874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17867924#comment-17867924 ]
Zhongmin Qiao commented on FLINK-35874:
---------------------------------------
Assign this pr to me please. https://github.com/apache/flink-cdc/pull/3488
PTAL [~Leonard], [~ruanhang1993]
> Check pureBinlogPhaseTables set before call getBinlogPosition method in BinlogSplitReader
> -----------------------------------------------------------------------------------------
>
> Key: FLINK-35874
> URL: https://issues.apache.org/jira/browse/FLINK-35874
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
> Reporter: Zhongmin Qiao
> Priority: Minor
> Attachments: image-2024-07-22-19-26-59-158.png, image-2024-07-22-19-27-19-366.png, image-2024-07-22-19-30-08-989.png, image-2024-07-22-19-36-20-481.png, image-2024-07-22-19-36-40-581.png, image-2024-07-22-19-37-35-542.png, image-2024-07-22-21-12-03-316.png
>
>
> The method getBinlogPosition of RecordUtil which is called by BinlogSplitReader.
> shouldEmit is a highly performance-consuming method. This is because it iterates through the sourceOffset map of the SourceRecord, and during the iteration, it also performs a toString() conversion on the value. Finally, it calls the putAll method of BinlogOffsetBuilder to put all the elements obtained from the iteration into the offsetMap (which involves another map traversal and hashcode computation). Despite the significant performance impact of getBinlogPosition, we still need to call it when emitting each DataChangeRecord, which reduces the efficiency of data processing in Flink CDC.
> !image-2024-07-22-19-26-59-158.png|width=545,height=222!
> !image-2024-07-22-19-27-19-366.png|width=545,height=119!
> However, we can optimize and avoid frequent invocations of getBinlogPosition by moving the check pureBinlogPhaseTables.contains(tableId) in the hasEnterPureBinlogPhase method before calling getBinlogPosition. This way, if the SourceRecord belongs to a pure binlog phase table, we can directly return true without the need for the highly performance-consuming getBinlogPosition method.
> diff
> !image-2024-07-22-21-12-03-316.png|width=548,height=236!
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)