You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "sivabalan narayanan (Jira)" <ji...@apache.org> on 2025/04/09 20:30:00 UTC
[jira] [Updated] (HUDI-8475) Generate all required stats required for MDT within DT write handles
[ https://issues.apache.org/jira/browse/HUDI-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
sivabalan narayanan updated HUDI-8475:
--------------------------------------
Remaining Estimate: 12h
Original Estimate: 12h
> Generate all required stats required for MDT within DT write handles
> --------------------------------------------------------------------
>
> Key: HUDI-8475
> URL: https://issues.apache.org/jira/browse/HUDI-8475
> Project: Apache Hudi
> Issue Type: Sub-task
> Components: metadata, writer-core
> Reporter: sivabalan narayanan
> Assignee: sivabalan narayanan
> Priority: Critical
> Fix For: 1.0.2
>
> Original Estimate: 12h
> Remaining Estimate: 12h
>
> As of now, some stats are sent to HoodieBackedTableMetadataWriter from the write handles, while some are generated by reading the base files or log on-demand within HoodieBackedTableMetadataWriter.update() call.
> With the new dag design, we wanted to populate all stats within the DT write handles only and send them back via WriteStatus.
> We do not plan to collect entire WriteStatus in the driver, but just the HoodieWRiteStat, and so we should be able to manage with WriteStatus holding all the required stats for all indexes in MDT.
>
> FILES: no additional work required
> col stats: generate all stats within write handles including base data files.
> bloom index: Better to ignore it here. We can do on-demand read from base files from within the MDTPartitioner.
> functional index stats: same as col stats. For bloom, we can defer.
> RLI: no additional work required.
> secondary index: lets generate all required stats from within all 3 write handles. for Append handle, we might have to read the entire file slice including the current file being written and stats have to be generated.
> partition stats: yet to design this.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)