[jira] [Updated] (HUDI-1554) Introduce buffering for streams in HUDI

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HUDI-1554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated HUDI-1554:
---------------------------------
    Labels: pull-request-available  (was: )

> Introduce buffering for streams in HUDI
> ---------------------------------------
>
>                 Key: HUDI-1554
>                 URL: https://issues.apache.org/jira/browse/HUDI-1554
>             Project: Apache Hudi
>          Issue Type: Improvement
>            Reporter: Prashant Wason
>            Assignee: Prashant Wason
>            Priority: Major
>              Labels: pull-request-available
>
> Input and Output streams created in HUDI through calls to HoodieWrapperFileSystem do not include any buffering unless the underlying file system implements buffering.
> DistributedFileSystem (over HDFS) does not implement any buffering. This leads to very large number of small-sized IO calls being send to the HDFS while performing HUDI IO operations like reading parquet, writing parquet, reading/writing log files, reading/writing instants, etc. 
> This patch introduces buffering at the HoodieWrapperFileSystem level so that all types of reads and writes benefit from buffering.
>  
> In my tests with at scale on HDFS writing 1million records into a parquet file (read from an existing parquet file in the same dataset), I observed the following benefits:
>  # about 40% reduction in total time to run the test  
>  # Total write calls to HDFS reduced from 19.1M -> 328
>  # Total read calls reduced from 229M -> 515K
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)