You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "Rajesh Mahindra (Jira)" <ji...@apache.org> on 2022/02/01 05:18:00 UTC

[jira] [Updated] (HUDI-3356) Conversion of write stats to metadata index records should use HoodieData throughout

     [ https://issues.apache.org/jira/browse/HUDI-3356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rajesh Mahindra updated HUDI-3356:
----------------------------------
    Sprint: Hudi-Sprint-Jan-31

> Conversion of write stats to metadata index records should use HoodieData throughout
> ------------------------------------------------------------------------------------
>
>                 Key: HUDI-3356
>                 URL: https://issues.apache.org/jira/browse/HUDI-3356
>             Project: Apache Hudi
>          Issue Type: Task
>          Components: writer-core
>            Reporter: Manoj Govindassamy
>            Assignee: Manoj Govindassamy
>            Priority: Blocker
>             Fix For: 0.11.0
>
>
> HoodieMetadataTableUtil convertMetadataToRecords() converts all write stats to metadata index records to List of HoodieRecords before passing on them to engine specific commit() to prep records. This can OOM driver. We need to use HoodieData<HoodieRecords> throughout. 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)