You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "steven zhang (Jira)" <ji...@apache.org> on 2020/11/17 10:45:00 UTC

[jira] [Created] (HUDI-1398) Align insert file size for reducing IO

steven zhang created HUDI-1398:
----------------------------------

             Summary: Align insert file size for reducing IO
                 Key: HUDI-1398
                 URL: https://issues.apache.org/jira/browse/HUDI-1398
             Project: Apache Hudi
          Issue Type: Improvement
            Reporter: steven zhang


currently we insert totalUnassignedInserts into new file if we have anything more records

and set number of new bucket records as follow:

recordsPerBucket.add(totalUnassignedInserts / insertBuckets); ([https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/UpsertPartitioner.java] L 188)

it just compute the avg records. and it may create new small file

for example:

totalUnassignedInserts = 250

insertRecordsPerBucket = 120

so insertBuckets = 3 (eg. file_a,file_b,file_c)

then  file_a = file_b = file_c = 83 

the small files will include above three file when next delta process

and we can reduce io by set file_a = 120 file_b = 120 file_c = 10 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)