[jira] [Commented] (HUDI-8420) Return all fields as mandatory for custom merging

["Y Ethan Guo (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HUDI-8420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17892094#comment-17892094 ] 

Y Ethan Guo commented on HUDI-8420:
-----------------------------------

After reviewing the HUDI-8203 PR, the following API in HoodieRecordMerger provides the functionality to return all fields:
{code:java}
/**
 * If false, whenever we have log files, we will need to read all columns
 * If true, mor merging can be done without all columns. The columns required can be configured
 * by overriding getMandatoryFieldsForMerging
 */
default boolean isProjectionCompatible() {
  return false;
} {code}

> Return all fields as mandatory for custom merging
> -------------------------------------------------
>
>                 Key: HUDI-8420
>                 URL: https://issues.apache.org/jira/browse/HUDI-8420
>             Project: Apache Hudi
>          Issue Type: Improvement
>            Reporter: Y Ethan Guo
>            Priority: Critical
>             Fix For: 1.1.0
>
>
> [https://github.com/apache/hudi/pull/11943#discussion_r1805674206]
> The default implementation of {{getMandatoryFieldsForMerging}} should return all columns since custom merging may need all columns regardless of the request schema. Only the {{DefaultSparkRecordMerger}} and {{{}OverwriteWithLatestSparkRecordMerger{}}}(and counterpart classes for other engines) should overwrite the implementation to return record key and optional preCombine field for better performance with projection.
>  
> Right now the user must provide {{getMandatoryFieldsForMerging}} to make sure the functionality is correct, which is OK as long as we update the docs (HUDI-8421).  For better dev experience, we should not require user to override this unless the user understands the tradeoff here.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)