[jira] [Resolved] (HUDI-1509) Major performance degradation due to rewriting records with default values

["sivabalan narayanan (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HUDI-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

sivabalan narayanan resolved HUDI-1509.
---------------------------------------
    Fix Version/s:     (was: 0.8.0)
                   0.7.0
       Resolution: Fixed

> Major performance degradation due to rewriting records with default values
> --------------------------------------------------------------------------
>
>                 Key: HUDI-1509
>                 URL: https://issues.apache.org/jira/browse/HUDI-1509
>             Project: Apache Hudi
>          Issue Type: Bug
>    Affects Versions: 0.6.0, 0.7.0, 0.8.0
>            Reporter: Prashant Wason
>            Assignee: Nishith Agarwal
>            Priority: Blocker
>              Labels: pull-request-available
>             Fix For: 0.7.0
>
>
> During the in-house testing for 0.5x to 0.6x release upgrade, I have detected a performance degradation for writes into HUDI. I have traced the issue due to the changes in the following commit
> [[HUDI-727]: Copy default values of fields if not present when rewriting incoming record with new schema|https://github.com/apache/hudi/commit/6d7ca2cf7e441ad19d32d7a25739e454f39ed253]
> I wrote a unit test to reduce the scope of testing as follows:
> # Take an existing parquet file from production dataset (size=690MB, #records=960K)
> # Read all the records from this parquet into a JavaRDD
> # Time the call HoodieWriteClient.bulkInsertPrepped(). (bulkInsertParallelism=1)
> The above scenario is directly taken from our production pipelines where each executor will ingest about a million record creating a single parquet file in a COW dataset. This is bulk insert only dataset.
> The time to complete the bulk insert prepped *decreased from 680seconds to 380seconds* when I reverted the above commit. 
> Schema details: This HUDI dataset uses a large schema with 51 fields in the record.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)