You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Prashant Wason (Jira)" <ji...@apache.org> on 2021/01/06 00:48:00 UTC
[jira] [Created] (HUDI-1509) Major performance degradation due to
rewriting records with default values
Prashant Wason created HUDI-1509:
------------------------------------
Summary: Major performance degradation due to rewriting records with default values
Key: HUDI-1509
URL: https://issues.apache.org/jira/browse/HUDI-1509
Project: Apache Hudi
Issue Type: Bug
Reporter: Prashant Wason
During the in-house testing for 0.5x to 0.6x release upgrade, I have detected a performance degradation for writes into HUDI. I have traced the issue due to the changes in the following commit
[[HUDI-727]: Copy default values of fields if not present when rewriting incoming record with new schema|https://github.com/apache/hudi/commit/6d7ca2cf7e441ad19d32d7a25739e454f39ed253]
I wrote a unit test to reduce the scope of testing as follows:
# Take an existing parquet file from production dataset (size=690MB, #records=960K)
# Read all the records from this parquet into a JavaRDD
# Time the call HoodieWriteClient.bulkInsertPrepped(). (bulkInsertParallelism=1)
The above scenario is directly taken from our production pipelines where each executor will ingest about a million record creating a single parquet file in a COW dataset. This is bulk insert only dataset.
The time to complete the bulk insert prepped *decreased from 680seconds to 380seconds* when I reverted the above commit.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)