You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "screamedddd (via GitHub)" <gi...@apache.org> on 2025/09/03 06:01:04 UTC
[I] [SUPPORT] spark write hudi mor table, enable async clustering and async compaction , have deltacommit FileNotFoundException [hudi]
screamedddd opened a new issue, #13826:
URL: https://github.com/apache/hudi/issues/13826
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
A clear and concise description of the problem.
**To Reproduce**
Steps to reproduce the behavior:
1. sparkStructed streaming write hudi mor table , after the program has been running for a while(maybe many hours) or kafka offset.reset = earliest , the program has a FileNotFoundException (deltacommit file not found) （maybe timeline check error?）
2. this is my enable clustering and compaction options
hoodie.metadata.index.column.stats.enable: false
clustering.schedule.enabled: true
hoodie.clustering.inline: false
hoodie.clustering.schedule.inline: false
hoodie.clustering.async.enabled: true
hoodie.compact.inline: false
hoodie.datasource.compaction.async.enable: true
hoodie.compact.inline.max.delta.commits: 10
hoodie.compact.schedule.inline: false
hoodie.clustering.plan.strategy.max.num.groups: 30
hoodie.clustering.plan.strategy.target.file.max.bytes: 1073741824
hoodie.clustering.plan.strategy.small.file.limit: 629145600
hoodie.clustering.updates.strategy: org.apache.hudi.client.clustering.update.strategy.SparkAllowUpdateStrategy
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version : 1.0.2
* Spark version : 3.3.2
* Hive version : 2.1.1
* Hadoop version : 3.0.0
* Storage (HDFS/S3/GCS..) : HDFS
* Running on Docker? (yes/no) : no
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
CompactionOperation{baseInstantTime='20250903111243844', dataFileCommitTime=Option{val=20250903111243844}, deltaFileNames=[.5c9337de-3148-4781-91ce-6e0b79988326-0_20250903111248773.log.1_188-9926-478112, .5c9337de-3148-4781-91ce-6e0b79988326-0_20250903111315002.log.1_172-9967-479828, .5c9337de-3148-4781-91ce-6e0b79988326-0_20250903111344327.log.1_185-9999-481519, .5c9337de-3148-4781-91ce-6e0b79988326-0_20250903111415158.log.1_177-10042-484078, .5c9337de-3148-4781-91ce-6e0b79988326-0_20250903111828795.log.1_183-10110-486260, .5c9337de-3148-4781-91ce-6e0b79988326-0_20250903111906942.log.1_189-10146-488116, .5c9337de-3148-4781-91ce-6e0b79988326-0_20250903111930043.log.1_188-10177-489832, .5c9337de-3148-4781-91ce-6e0b79988326-0_20250903112011162.log.1_190-10208-491546, .5c9337de-3148-4781-91ce-6e0b79988326-0_20250903112058908.log.1_185-10241-493252, .5c9337de-3148-4781-91ce-6e0b79988326-0_20250903112118717.log.1_195-10276-495006, .5c9337de-3148-4781-91ce-6e0b79988326-0_202509031121453
14.log.1_194-10395-498945, .5c9337de-3148-4781-91ce-6e0b79988326-0_20250903112428515.log.1_189-10446-500951, .5c9337de-3148-4781-91ce-6e0b79988326-0_20250903112513606.log.1_184-10477-502486, .5c9337de-3148-4781-91ce-6e0b79988326-0_20250903112551782.log.1_191-10509-504006, .5c9337de-3148-4781-91ce-6e0b79988326-0_20250903112611624.log.1_192-10542-505743, .5c9337de-3148-4781-91ce-6e0b79988326-0_20250903112757009.log.1_191-10667-510290, .5c9337de-3148-4781-91ce-6e0b79988326-0_20250903112917908.log.1_192-10747-512648, .5c9337de-3148-4781-91ce-6e0b79988326-0_20250903112952298.log.1_198-10779-514392, .5c9337de-3148-4781-91ce-6e0b79988326-0_20250903113035020.log.1_192-10811-516136, .5c9337de-3148-4781-91ce-6e0b79988326-0_20250903113055089.log.1_203-10843-517882, .5c9337de-3148-4781-91ce-6e0b79988326-0_20250903113114665.log.1_196-10878-519762, .5c9337de-3148-4781-91ce-6e0b79988326-0_20250903113155197.log.1_192-10911-521515, .5c9337de-3148-4781-91ce-6e0b79988326-0_20250903113232459.log.1_192-
10944-523258, .5c9337de-3148-4781-91ce-6e0b79988326-0_20250903113306666.log.1_175-10977-524965, .5c9337de-3148-4781-91ce-6e0b79988326-0_20250903113344098.log.1_195-11010-526532], dataFileName=Option{val=5c9337de-3148-4781-91ce-6e0b79988326-0_10-10698-510315_20250903111243844.parquet}, id='HoodieFileGroupId{partitionPath='20250601', fileId='5c9337de-3148-4781-91ce-6e0b79988326-0'}', metrics={TOTAL_LOG_FILES=25.0, TOTAL_IO_READ_MB=1.0, TOTAL_LOG_FILES_SIZE=300754.0, TOTAL_IO_WRITE_MB=1.0, TOTAL_IO_MB=2.0}, bootstrapFilePath=Optional.empty}] files
25/09/03 11:48:44 ERROR AsyncCompactService: Compactor executor failed
org.apache.hudi.exception.HoodieCompactionException: Could not compact hdfs://ns2/prod/hudi/ods/table01
at org.apache.hudi.table.action.compact.RunCompactionActionExecutor.execute(RunCompactionActionExecutor.java:130)
at org.apache.hudi.table.HoodieSparkMergeOnReadTable.compact(HoodieSparkMergeOnReadTable.java:155)
at org.apache.hudi.client.BaseHoodieTableServiceClient.compact(BaseHoodieTableServiceClient.java:320)
at org.apache.hudi.client.BaseHoodieWriteClient.compact(BaseHoodieWriteClient.java:1175)
at org.apache.hudi.client.BaseHoodieWriteClient.compact(BaseHoodieWriteClient.java:1095)
at org.apache.hudi.client.HoodieSparkCompactor.compact(HoodieSparkCompactor.java:50)
at org.apache.hudi.async.AsyncCompactService.lambda$null$0(AsyncCompactService.java:83)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hudi.exception.HoodieIOException: Could not read commit details from hdfs://ns2/prod/hudi/ods/table01/.hoodie/timeline/20250903112757009_20250903112905847.deltacommit
at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.readDataStreamFromPath(ActiveTimelineV2.java:728)
at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.getContentStream(ActiveTimelineV2.java:271)
at org.apache.hudi.common.table.timeline.BaseHoodieTimeline.getInstantContentStream(BaseHoodieTimeline.java:558)
at org.apache.hudi.common.table.timeline.HoodieTimeline.readInstantContent(HoodieTimeline.java:138)
at org.apache.hudi.common.table.timeline.HoodieTimeline.readCommitMetadata(HoodieTimeline.java:165)
at org.apache.hudi.common.table.TableSchemaResolver.lambda$getTableInternalSchemaFromCommitMetadata$8(TableSchemaResolver.java:349)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
at org.apache.hudi.common.table.timeline.BaseHoodieTimeline.<init>(BaseHoodieTimeline.java:93)
at org.apache.hudi.common.table.timeline.versioning.v2.BaseTimelineV2.<init>(BaseTimelineV2.java:43)
at org.apache.hudi.common.table.timeline.versioning.v2.BaseTimelineV2.<init>(BaseTimelineV2.java:39)
at org.apache.hudi.common.table.timeline.versioning.v2.TimelineV2Factory.createDefaultTimeline(TimelineV2Factory.java:44)
at org.apache.hudi.common.table.timeline.BaseHoodieTimeline.filter(BaseHoodieTimeline.java:321)
at org.apache.hudi.common.table.TableSchemaResolver.getTableInternalSchemaFromCommitMetadata(TableSchemaResolver.java:346)
at org.apache.hudi.table.SparkBroadcastManager.getSchemaEvolutionConfigs(SparkBroadcastManager.java:154)
at org.apache.hudi.table.SparkBroadcastManager.prepareAndBroadcast(SparkBroadcastManager.java:95)
at org.apache.hudi.common.util.Option.ifPresent(Option.java:101)
at org.apache.hudi.table.action.compact.HoodieCompactor.compact(HoodieCompactor.java:157)
at org.apache.hudi.table.action.compact.RunCompactionActionExecutor.execute(RunCompactionActionExecutor.java:108)
... 10 more
Caused by: java.io.FileNotFoundException: File does not exist: /prod/hudi/ods/table01/.hoodie/timeline/20250903112757009_20250903112905847.deltacommit
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:85)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:75)
at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:152)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1909)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:735)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:415)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:902)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:889)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:878)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1046)
at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:340)
at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:336)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:353)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:976)
at org.apache.hudi.hadoop.fs.HoodieWrapperFileSystem.open(HoodieWrapperFileSystem.java:481)
at org.apache.hudi.storage.hadoop.HoodieHadoopStorage.open(HoodieHadoopStorage.java:149)
at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.readDataStreamFromPath(ActiveTimelineV2.java:726)
... 33 more
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /prod/hudi/ods/table01/.hoodie/timeline/20250903112757009_20250903112905847.deltacommit
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:85)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:75)
at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:152)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1909)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:735)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:415)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1612)
at org.apache.hadoop.ipc.Client.call(Client.java:1558)
at org.apache.hadoop.ipc.Client.call(Client.java:1455)
at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:242)
at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:129)
at com.sun.proxy.$Proxy39.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:333)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy40.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:900)
... 44 more
25/09/03 11:48:44 INFO HoodieStreamingSink: Async Compactor shutdown. Errored ? true
25/09/03 11:48:44 INFO SparkContext: Starting job: start at WriteHudiSparkWithConfigTable.scala:122
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] spark write hudi mor table, enable async clustering and async compaction , have deltacommit FileNotFoundException [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #13826:
URL: https://github.com/apache/hudi/issues/13826#issuecomment-3457288297
Hi @screamedddd
Is this issue still persisting? If possible, we kindly request you provide a minimal reproducible code example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] spark write hudi mor table, enable async clustering and async compaction , have deltacommit FileNotFoundException [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #13826:
URL: https://github.com/apache/hudi/issues/13826#issuecomment-3815393505
Hi @screamedddd
Could you please provide a minimal reproducible example? This will help us move forward with the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org