You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "sivabalan narayanan (Jira)" <ji...@apache.org> on 2021/01/29 00:00:00 UTC
[jira] [Comment Edited] (HUDI-1278) Need a generic payload class
which can skip late arriving data based on specific fields
[ https://issues.apache.org/jira/browse/HUDI-1278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17270965#comment-17270965 ]
sivabalan narayanan edited comment on HUDI-1278 at 1/28/21, 11:59 PM:
----------------------------------------------------------------------
[~shenhong] : Can you clarify the requirements for this ? Is it different from what the newly introduced [payload|https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/model/DefaultHoodieRecordPayload.java] can do.
was (Author: shivnarayan):
[~vbalaji]: Can you clarify the requirements for this ? Is it different from what the newly introduced [payload|https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/model/DefaultHoodieRecordPayload.java] can do.
> Need a generic payload class which can skip late arriving data based on specific fields
> ---------------------------------------------------------------------------------------
>
> Key: HUDI-1278
> URL: https://issues.apache.org/jira/browse/HUDI-1278
> Project: Apache Hudi
> Issue Type: New Feature
> Components: DeltaStreamer, Spark Integration
> Reporter: Balaji Varadarajan
> Assignee: shenh062326
> Priority: Major
> Labels: user-support-issues
> Fix For: 0.8.0
>
>
> Context : [https://lists.apache.org/thread.html/rd5d805d29c2f704d8ff2729457d27bca42e890bc01fc8e5e1f1943e3%40%3Cdev.hudi.apache.org%3E]
> We need to implement a Payload class (like OverwriteWithLatestAvroPayload) which will skip late arriving data.
> Notes:
> # combineAndGetUpdateValue() would need work
> # The ordering needs to be specified based on 1 or more fields and should be configurable.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)