You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Ethan Guo (Jira)" <ji...@apache.org> on 2024/03/29 01:26:00 UTC
[jira] [Closed] (HUDI-2923) Unable to read from metadata table when a compaction is in progress or has failed
[ https://issues.apache.org/jira/browse/HUDI-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ethan Guo closed HUDI-2923.
---------------------------
Resolution: Fixed
> Unable to read from metadata table when a compaction is in progress or has failed
> ---------------------------------------------------------------------------------
>
> Key: HUDI-2923
> URL: https://issues.apache.org/jira/browse/HUDI-2923
> Project: Apache Hudi
> Issue Type: Bug
> Components: compaction, table-service
> Reporter: Prashant Wason
> Assignee: Prashant Wason
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 0.10.0
>
>
> When reading from metadata table, the [readers are opened with the latest file slices|[https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java#L248].]
> When a compaction is in progress, the latest file slice does not have any base-file or log file (yet). Hence we are unable to read data from the metadata table.
>
> There are two cases here:
> # Compaction eventually completes: We will be able to read data from the metadata table.
> # Compaction fails: We will not be able to read data unless the next time compaction runs. This can be a fatal issue if the next writer tries to perform an update which requires listing partition from the metadata table.
>
> Relevant logs from a unit test:
>
> 13084 [main] INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView - *Pending Compaction instant for* (FileSlice \{fileGroupId=HoodieFileGroupId{partitionPath='files', fileId='2733a6ef-4bfd-444d-91bd-b42c3b66a84e-0'}, baseCommitTime=002001, baseFile='null', logFiles='[]'}) is :Option\{val=(002001,CompactionOperation{baseInstantTime='001', dataFileCommitTime=Option{val=001}, deltaFileNames=[.2733a6ef-4bfd-444d-91bd-b42c3b66a84e-0_001.log.1_0-34-36], dataFileName=Option\{val=2733a6ef-4bfd-444d-91bd-b42c3b66a84e-0_0-16-20_001.hfile}, id='HoodieFileGroupId\{partitionPath='files', fileId='2733a6ef-4bfd-444d-91bd-b42c3b66a84e-0'}', metrics={}, bootstrapFilePath=Optional.empty})}
> 13084 [main] INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView - File Slice (FileSlice \{fileGroupId=HoodieFileGroupId{partitionPath='files', fileId='2733a6ef-4bfd-444d-91bd-b42c3b66a84e-0'}, *baseCommitTime=002001, baseFile='null', logFiles='[]'}) is in pending compaction*
>
> 13089 [main] INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner - *Number of log files scanned => 0*
> 13089 [main] INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner - MaxMemoryInBytes allowed for compaction => 0
> 13089 [main] INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner - Number of entries in MemoryBasedMap in ExternalSpillableMap => 0
> 13089 [main] INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner - Total size in bytes of MemoryBasedMap in ExternalSpillableMap => 0
> 13089 [main] INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner - Number of entries in DiskBasedMap in ExternalSpillableMap => 0
> 13089 [main] INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner - Size of file spilled to disk => 0
> 13089 [main] INFO org.apache.hudi.metadata.HoodieBackedTableMetadata - *Opened metadata log files from []* at instant (dataset instant=002, metadata instant=002) in 2 ms
> 13089 [main] INFO org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata read for key __all_partitions__ took [baseFileRead, logMerge] [0, 0] ms
> 13090 [main] INFO org.apache.hudi.metadata.BaseTableMetadata - *Listed partitions from metadata: #partitions=0*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)