You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Ruan Hang (Jira)" <ji...@apache.org> on 2024/11/20 11:04:00 UTC
[jira] [Resolved] (FLINK-36750) Paimon connector would reuse sequence number when schema evolution happened
[ https://issues.apache.org/jira/browse/FLINK-36750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ruan Hang resolved FLINK-36750.
-------------------------------
Resolution: Fixed
> Paimon connector would reuse sequence number when schema evolution happened
> ---------------------------------------------------------------------------
>
> Key: FLINK-36750
> URL: https://issues.apache.org/jira/browse/FLINK-36750
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
> Affects Versions: cdc-3.2.0
> Reporter: Yanquan Lv
> Assignee: Yanquan Lv
> Priority: Major
> Labels: pull-request-available
> Fix For: cdc-3.3.0, cdc-3.2.1
>
> Attachments: image-2024-11-20-13-00-58-282.png, image-2024-11-20-13-02-47-612.png, image-2024-11-20-13-04-53-635.png
>
>
> When schema evolution happened, we will prepare commit and recreate a new FileStoreWrite to obtain the latest schema. However, FileStoreWrite maintain some information like sequence number in memory, we can't directly remove and recreate one FileStoreWrite, instead, we should extract the information of Write and rebuild with this information.
> The sequence number is used to determine the order of data with two identical primary keys, If we don't strictly maintain this order, it may lead to unexpected situations.
> The following picture show The problem we are currently facing：
> 1) Schema evolution happened between the second and third files(`{*}schema_id{*}` changed)
> !image-2024-11-20-13-04-53-635.png!
> 2）The expected sequence number here should be increasing, however, there is an overlap of `{*}min_sequence_number{*}` between the third file and the second file.
> !image-2024-11-20-13-02-47-612.png!
> Due to the confusion of sequence numbers, we may read the data of update-before.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)