[jira] [Closed] (HUDI-4094) Allow bulk insert partitioner to specify the fileID prefixes to use

["Prashant Wason (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HUDI-4094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Prashant Wason closed HUDI-4094.
--------------------------------
    Resolution: Duplicate

> Allow bulk insert partitioner to specify the fileID prefixes to use
> -------------------------------------------------------------------
>
>                 Key: HUDI-4094
>                 URL: https://issues.apache.org/jira/browse/HUDI-4094
>             Project: Apache Hudi
>          Issue Type: New Feature
>            Reporter: Prashant Wason
>            Assignee: Prashant Wason
>            Priority: Major
>
> This is useful for using bulk insert when bootstrapping metadata table indexes.
> Currently we use upsertPrepped to write to metadata table. The upsert code path is not optimized for very large writes (1Billion+ records) due to the work load profiling and upsert partitioning overheads. 
> Bulk insert for metadata table requires the partitions to be written to files which have special names and hence random fileIDs cannot be used (as currently implemented).



--
This message was sent by Atlassian Jira
(v8.20.7#820007)