[jira] [Closed] (HUDI-3365) Make sure Metadata Records always bear full file-size

["Alexey Kudinkin (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HUDI-3365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexey Kudinkin closed HUDI-3365.
---------------------------------
    Resolution: Fixed

> Make sure Metadata Records always bear full file-size
> -----------------------------------------------------
>
>                 Key: HUDI-3365
>                 URL: https://issues.apache.org/jira/browse/HUDI-3365
>             Project: Apache Hudi
>          Issue Type: Bug
>            Reporter: Alexey Kudinkin
>            Assignee: Alexey Kudinkin
>            Priority: Blocker
>              Labels: HUDI-bug, pull-request-available
>             Fix For: 0.11.0
>
>
> Currently, when log-file is appended the Metadata Table record w/ the size of appended delta will be submitted to MT. 
> MT, in turn, will sum it up with whatever record is currently persisted there at the moment (REF: [https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java#L227)]
>  
> This is problematic in multiple ways:
>  # We're abusing FileStatus interface which unequivocally relates the size as a full file-size.
>  # MT receiving new record can't determine whether it's received a delta or a new record that have to override the old one. As such, it stick to the protocol, that if record already exists it will treat new one as a delta.
> This behavior is very implicit, and there are currently at least one bug where actually full file-size is provided, while it would lead to the incorrect file size being stored in MT.
>  
> Proposal: Unify the data flow and always provide full, up-to-date file-size to the MT. Even in the log-file appending flow, we do have a way to reconstruct full file-size (from `AppendResult`, w/o additional `getFileStatus` necessary
>  
> Currently, when log-file is appended (on FS that supports it), only the appended delta will be submitted w/in records to MT



--
This message was sent by Atlassian Jira
(v8.20.1#820001)