You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Y Ethan Guo (Jira)" <ji...@apache.org> on 2025/01/07 20:07:00 UTC
[jira] [Assigned] (HUDI-8661) Support log compaction in HoodieSparkFileGroupReaderBasedMergeHandle
[ https://issues.apache.org/jira/browse/HUDI-8661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Y Ethan Guo reassigned HUDI-8661:
---------------------------------
Assignee: Y Ethan Guo
> Support log compaction in HoodieSparkFileGroupReaderBasedMergeHandle
> --------------------------------------------------------------------
>
> Key: HUDI-8661
> URL: https://issues.apache.org/jira/browse/HUDI-8661
> Project: Apache Hudi
> Issue Type: Sub-task
> Reporter: Y Ethan Guo
> Assignee: Y Ethan Guo
> Priority: Blocker
> Fix For: 1.0.1
>
> Original Estimate: 8h
> Time Spent: 1h
> Remaining Estimate: 7h
>
> The new file group reader-based compaction in HoodieSparkFileGroupReaderBasedMergeHandle has not supported log compaction yet. Right now it's disabled through the code below in HoodieCompactor. When enabled, there are a few tests that fail: TestHoodieMergeOnReadTable.testLogBlocksCountsAfterLogCompaction, TestHoodieClientOnMergeOnReadStorage.testArchivalOnLogCompaction.
> {code:java}
> boolean useFileGroupReaderBasedCompaction = context.supportsFileGroupReader() // the engine needs to support fg reader first
> && !metaClient.isMetadataTable()
> && config.getBooleanOrDefault(HoodieReaderConfig.FILE_GROUP_READER_ENABLED)
> && operationType == WriteOperationType.COMPACT
> && !hasBootstrapFile(operations) // bootstrap file read for fg reader is not ready
> && StringUtils.isNullOrEmpty(config.getInternalSchema()) // schema evolution support for fg reader is not ready
> && config.populateMetaFields(); // Virtual key support by fg reader is not ready
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)