You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Yu Li (Jira)" <ji...@apache.org> on 2022/09/08 04:47:00 UTC
[jira] [Commented] (FLINK-28910) CDC From Mysql To Hbase Bugs
[ https://issues.apache.org/jira/browse/FLINK-28910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17601628#comment-17601628 ]
Yu Li commented on FLINK-28910:
-------------------------------
From my point of view, a better solution is to facilitate the atomic operations in HBase (Table#checkAndMutate or Table#muateRow) for the update-alike change data ingestion. However, currently Flink's `HBaseSinkFunction` takes usage of HBase's `BufferedMutator` and `BufferedMutator` only supports `Mutation` (actually [currently|https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutator.java#L93-L99] only Put and Delete), and operations like `RowMutations` or `CheckAndMuate` are not supported. Let me check whether we could do something on the HBase side.
cc [~zhangduo]
> CDC From Mysql To Hbase Bugs
> ----------------------------
>
> Key: FLINK-28910
> URL: https://issues.apache.org/jira/browse/FLINK-28910
> Project: Flink
> Issue Type: Bug
> Components: Connectors / HBase
> Reporter: TE
> Priority: Major
> Labels: pull-request-available, stale-blocker
>
> I use Flink for CDC from Mysql to Hbase.
> The problem I encountered is that the Mysql record is updated (not deleted), but the record in hbase is deleted sometimes.
> I tried to analyze the problem and found the reason as follows:
> The update action of Mysql will be decomposed into delete + insert by Flink.
> The Hbase connector uses a mutator to handle this set of actions.
> However, if the order of these actions is not actively set, the processing of the mutator will not guarantee the order of execution.
> Therefore, when the update of Mysql is triggered, it is possible that hbase actually performed the actions in the order of put + delete, resulting in the data being deleted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)