You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Raymond Xu (Jira)" <ji...@apache.org> on 2022/05/03 09:09:00 UTC
[jira] [Updated] (HUDI-3453) Metadata table throws NPE when scheduling compaction plan
[ https://issues.apache.org/jira/browse/HUDI-3453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raymond Xu updated HUDI-3453:
-----------------------------
Sprint: Hudi-Sprint-Mar-07, Hudi-Sprint-Mar-14, Hudi-Sprint-Mar-21, Hudi-Sprint-Mar-22, Hudi-Sprint-Apr-05, Hudi-Sprint-Apr-19, Hudi-Sprint-Apr-25, Hudi-Sprint-May-02 (was: Hudi-Sprint-Mar-07, Hudi-Sprint-Mar-14, Hudi-Sprint-Mar-21, Hudi-Sprint-Mar-22, Hudi-Sprint-Apr-05, Hudi-Sprint-Apr-19, Hudi-Sprint-Apr-25)
> Metadata table throws NPE when scheduling compaction plan
> ---------------------------------------------------------
>
> Key: HUDI-3453
> URL: https://issues.apache.org/jira/browse/HUDI-3453
> Project: Apache Hudi
> Issue Type: Bug
> Components: core
> Reporter: Danny Chen
> Assignee: Yue Zhang
> Priority: Critical
> Labels: pull-request-available
> Fix For: 0.12.0
>
>
> {code:java}
> Caused by: org.apache.hudi.exception.HoodieException: Error occurs when executing flatMap
> at org.apache.hudi.common.function.FunctionWrapper.lambda$throwingFlatMapWrapper$1(FunctionWrapper.java:50)
> at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:269)
> at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:747)
> at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:721)
> at java.util.stream.AbstractTask.compute(AbstractTask.java:327)
> at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
> at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> at java.util.concurrent.ForkJoinPool.helpComplete(ForkJoinPool.java:1870)
> at java.util.concurrent.ForkJoinPool.externalHelpComplete(ForkJoinPool.java:2467)
> at java.util.concurrent.ForkJoinTask.externalAwaitDone(ForkJoinTask.java:324)
> at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:405)
> at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734)
> at java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:714)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
> at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
> at org.apache.hudi.client.common.HoodieFlinkEngineContext.flatMap(HoodieFlinkEngineContext.java:136)
> at org.apache.hudi.table.action.compact.HoodieCompactor.generateCompactionPlan(HoodieCompactor.java:263)
> at org.apache.hudi.table.action.compact.ScheduleCompactionActionExecutor.scheduleCompaction(ScheduleCompactionActionExecutor.java:122)
> at org.apache.hudi.table.action.compact.ScheduleCompactionActionExecutor.execute(ScheduleCompactionActionExecutor.java:92)
> at org.apache.hudi.table.HoodieFlinkMergeOnReadTable.scheduleCompaction(HoodieFlinkMergeOnReadTable.java:109)
> at org.apache.hudi.client.AbstractHoodieWriteClient.scheduleTableServiceInternal(AbstractHoodieWriteClient.java:1100)
> at org.apache.hudi.client.AbstractHoodieWriteClient.scheduleTableService(AbstractHoodieWriteClient.java:1083)
> at org.apache.hudi.client.AbstractHoodieWriteClient.scheduleCompactionAtInstant(AbstractHoodieWriteClient.java:850)
> at org.apache.hudi.client.AbstractHoodieWriteClient.scheduleCompaction(AbstractHoodieWriteClient.java:841)
> at org.apache.hudi.util.CompactionUtil.scheduleCompaction(CompactionUtil.java:64)
> at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$notifyCheckpointComplete$2(StreamWriteOperatorCoordinator.java:229)
> at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$execute$0(NonThrownExecutor.java:93)
> ... 3 more
> Caused by: org.apache.hudi.exception.HoodieMetadataException: Failed to retrieve files in partition oss://datalake-huifu/hudi/poc/ods/pnrweb_prod/trans_log/20220216 from metadata
> at org.apache.hudi.metadata.BaseTableMetadata.getAllFilesInPartition(BaseTableMetadata.java:124)
> at org.apache.hudi.metadata.HoodieMetadataFileSystemView.listPartition(HoodieMetadataFileSystemView.java:65)
> at org.apache.hudi.common.table.view.AbstractTableFileSystemView.lambda$ensurePartitionLoadedCorrectly$9(AbstractTableFileSystemView.java:304)
> at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
> at org.apache.hudi.common.table.view.AbstractTableFileSystemView.ensurePartitionLoadedCorrectly(AbstractTableFileSystemView.java:295)
> at org.apache.hudi.common.table.view.AbstractTableFileSystemView.getLatestFileSlices(AbstractTableFileSystemView.java:578)
> at org.apache.hudi.common.table.view.PriorityBasedFileSystemView.execute(PriorityBasedFileSystemView.java:80)
> at org.apache.hudi.common.table.view.PriorityBasedFileSystemView.getLatestFileSlices(PriorityBasedFileSystemView.java:170)
> at org.apache.hudi.table.action.compact.HoodieCompactor.lambda$generateCompactionPlan$30498406$1(HoodieCompactor.java:264)
> at org.apache.hudi.common.function.FunctionWrapper.lambda$throwingFlatMapWrapper$1(FunctionWrapper.java:48)
> ... 32 more
> Caused by: java.lang.NullPointerException
> at org.apache.hudi.io.storage.HoodieHFileReader.getRecordByKey(HoodieHFileReader.java:249)
> at org.apache.hudi.io.storage.HoodieFileReader.getRecordByKey(HoodieFileReader.java:49)
> at org.apache.hudi.metadata.HoodieBackedTableMetadata.readFromBaseAndMergeWithLogRecords(HoodieBackedTableMetadata.java:195)
> at org.apache.hudi.metadata.HoodieBackedTableMetadata.getRecordsByKeys(HoodieBackedTableMetadata.java:141)
> at org.apache.hudi.metadata.HoodieBackedTableMetadata.getRecordByKey(HoodieBackedTableMetadata.java:124)
> at org.apache.hudi.metadata.BaseTableMetadata.fetchAllFilesInPartition(BaseTableMetadata.java:184)
> at org.apache.hudi.metadata.BaseTableMetadata.getAllFilesInPartition(BaseTableMetadata.java:122)
> ... 41 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)