[jira] [Comment Edited] (HUDI-1258) Small file handling Merges can be handled without actual merging

["Raymond Xu (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HUDI-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17457244#comment-17457244 ] 

Raymond Xu edited comment on HUDI-1258 at 12/10/21, 4:59 PM:
-------------------------------------------------------------

[~vinoth] After going through the code, had some ideas for the solution. Just to confirm my understanding, we probably need to do these 2
 * in org.apache.hudi.table.action.commit.UpsertPartitioner#assignInserts, record a count of updates before inserts will be appended to the updating small files; this can be saved in bucket info from the upsert partitioner
 * in org.apache.hudi.io.HoodieMergeHandle#init, when initiate keyToNewRecords, stop adding record from the iterator when reaching the count. Inserts should be saved to a separate "ExternalSpillableList" instead and processed after 

 

Update: as discussed, it makes sense to error out during upsert when duplicate-key records are seen in merge handle


was (Author: xushiyan):
[~vinoth] After going through the code, had some ideas for the solution. Just to confirm my understanding, we probably need to do these 2
 * in org.apache.hudi.table.action.commit.UpsertPartitioner#assignInserts, record a count of updates before inserts will be appended to the updating small files; this can be saved in bucket info from the upsert partitioner
 * in org.apache.hudi.io.HoodieMergeHandle#init, when initiate keyToNewRecords, stop adding record from the iterator when reaching the count. Inserts should be saved to a separate "ExternalSpillableList" instead and processed after 

> Small file handling Merges can be handled without actual merging
> ----------------------------------------------------------------
>
>                 Key: HUDI-1258
>                 URL: https://issues.apache.org/jira/browse/HUDI-1258
>             Project: Apache Hudi
>          Issue Type: Improvement
>          Components: Writer Core
>    Affects Versions: 0.9.0
>            Reporter: Vinoth Chandar
>            Assignee: Raymond Xu
>            Priority: Blocker
>             Fix For: 0.11.0
>
>
> If a file slice gets inserts into MergeHandle, for file sizing reasons, there is no reason to really build the hashmap and merge. 
>  
> This will also avoid the issue of insert with the same duplicate key overwriting the previous value 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)