You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "sivabalan narayanan (Jira)" <ji...@apache.org> on 2024/11/04 23:58:00 UTC
[jira] [Updated] (HUDI-8474) Design and Impl MDT repartitioner to assist with writing to MDT
[ https://issues.apache.org/jira/browse/HUDI-8474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
sivabalan narayanan updated HUDI-8474:
--------------------------------------
Description:
We need a repartitoner for MDT where we take in HoodieData<HoodieRecords> and return 1 spark task pertaining to 1 file slice in MDT.
For eg, for FILES, its typically 1 file slice.
for col stats, RLI, etc its based on how user has configured it.
We should be doing sort within partitioner as well since w/ hfile we might have to sort the keys.
Except partition stats index, every other index should be straight forward. For partition stats record generation, we have a tracking ticket https://issues.apache.org/jira/browse/HUDI-8476
was:
We need a repartitoner for MDT where we take in HoodieData<HoodieRecords> and return 1 spark task pertaining to 1 file slice in MDT.
For eg, for FILES, its typically 1 file slice.
for col stats, RLI, etc its based on how user has configured it.
We should be doing sort within partitioner as well since w/ hfile we might have to sort the keys.
> Design and Impl MDT repartitioner to assist with writing to MDT
> ---------------------------------------------------------------
>
> Key: HUDI-8474
> URL: https://issues.apache.org/jira/browse/HUDI-8474
> Project: Apache Hudi
> Issue Type: Sub-task
> Components: metadata, writer-core
> Reporter: sivabalan narayanan
> Assignee: sivabalan narayanan
> Priority: Major
> Fix For: 1.0.0
>
>
> We need a repartitoner for MDT where we take in HoodieData<HoodieRecords> and return 1 spark task pertaining to 1 file slice in MDT.
> For eg, for FILES, its typically 1 file slice.
> for col stats, RLI, etc its based on how user has configured it.
> We should be doing sort within partitioner as well since w/ hfile we might have to sort the keys.
>
> Except partition stats index, every other index should be straight forward. For partition stats record generation, we have a tracking ticket https://issues.apache.org/jira/browse/HUDI-8476
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)