You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Ethan Guo (Jira)" <ji...@apache.org> on 2024/06/10 06:57:00 UTC

[jira] [Closed] (HUDI-6798) Implement event-time-based merging mode in FileGroupReader

     [ https://issues.apache.org/jira/browse/HUDI-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ethan Guo closed HUDI-6798.
---------------------------
    Resolution: Fixed

> Implement event-time-based merging mode in FileGroupReader
> ----------------------------------------------------------
>
>                 Key: HUDI-6798
>                 URL: https://issues.apache.org/jira/browse/HUDI-6798
>             Project: Apache Hudi
>          Issue Type: New Feature
>            Reporter: Ethan Guo
>            Assignee: Ethan Guo
>            Priority: Blocker
>              Labels: hudi-1.0.0-beta2, pull-request-available
>             Fix For: 1.0.0
>
>
> To achieve this, we should add a new table config {{hoodie.record.merge.mode}} to control the record merging mode and behavior in the new file group reader ({{{}HoodieFileGroupReader{}}}) and implements event-time ordering in it. The table config {{hoodie.record.merge.mode}} is going to be the single config that determines how the record merging happens in release 1.0 and beyond.
>  
> Three merging modes to define:
>  * {{{}OVERWRITE_WITH_LATEST{}}}: using transaction time to merge records, i.e., the record from later transaction overwrites the earlier record with the same key. This corresponds to the behavior of existing payload class {{{}OverwriteWithLatestAvroPayload{}}}.
>  * {{{}EVENT_TIME_ORDERING{}}}: using event time as the ordering to merge records, i.e., the record with the larger event time overwrites the record with the smaller event time on the same key, regardless of transaction time. The event time or preCombine field needs to be specified by the user. This corresponds to the behavior of existing payload class {{{}DefaultHoodieRecordPayload{}}}.
>  * {{{}CUSTOM{}}}: using custom merging logic specified by the user. When a user specifies a custom record merger strategy or payload class with Avro record merger, this is going to be specified so the record merging follows user-defined logic as before.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)