You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "Y Ethan Guo (Jira)" <ji...@apache.org> on 2024/09/23 18:56:00 UTC

[jira] [Commented] (HUDI-8142) Multi-writer unique key enforcement for OCC/NBCC

    [ https://issues.apache.org/jira/browse/HUDI-8142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17883997#comment-17883997 ] 

Y Ethan Guo commented on HUDI-8142:
-----------------------------------

Deferring this to Hudi 1.1.0 since the multi-writer unique key enforcement for inserts require fundamental redesign on concurrency control and indexing.

> Multi-writer unique key enforcement for OCC/NBCC
> ------------------------------------------------
>
>                 Key: HUDI-8142
>                 URL: https://issues.apache.org/jira/browse/HUDI-8142
>             Project: Apache Hudi
>          Issue Type: Task
>            Reporter: Ethan Guo (this is the old account; please use "yihua")
>            Assignee: Danny Chen
>            Priority: Critical
>             Fix For: 1.1.0
>
>
> When multiple writers try to insert records of the same record key that does not exsit in the table, both writers can succeed leading to duplicates in the table.  In reality, this might not happen in common use cases (e.g., streaming CDC logs to table through Kafka).  Still, if we'd like to strictly guarantee key uniqueness for concurrent inserts, we may need the Hudi index to be aware of multiple writers (right now the index only reads committed data), so that we can efficiently identify the same record key across multiple writers.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)