You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/04/13 09:27:57 UTC

[GitHub] [iceberg] wndemon opened a new issue, #4550: the snapshot file is lost when write iceberg using flink Failed to open input stream for file File does not exist

wndemon opened a new issue, #4550:
URL: https://github.com/apache/iceberg/issues/4550

   通过iceberg-flink-runtime-1.12-0.13.1.jar 使用flink写iceberg,偶尔会出现下述问题,快照文件丢失,找不到快照文件
   
     write iceberg from flink-runtime-1.12-0.13.1.jar, the following problems occasionally occur: the snapshot file is lost and cannot be found
   
   2022-04-13 09:54:48,657 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - flink_fae_iface_ma_details_header-IcebergFilesCommitter -> Sink: flink_fae_iface_ma_details_header-IcebergSink hdfs://hdfs/user/iceberg_catalog/db__fea_fae_iceberg/flink_fae_iface_ma_details_header (1/1) (09b19ba94ff66428538bb4bb3dae372b) switched from RUNNING to FAILED on 192.168.3.221:6122-088419 @ 192.168.3.221 (dataPort=42400).
   java.lang.NullPointerException: null
   	at org.apache.iceberg.util.LockManagers$InMemoryLockManager.release(LockManagers.java:225) ~[?:?]
   	at org.apache.iceberg.hadoop.HadoopTableOperations.renameToFinal(HadoopTableOperations.java:374) ~[?:?]
   	at org.apache.iceberg.hadoop.HadoopTableOperations.commit(HadoopTableOperations.java:158) ~[?:?]
   	at org.apache.iceberg.SnapshotProducer.lambda$commit$2(SnapshotProducer.java:300) ~[?:?]
   	at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:404) ~[?:?]
   	at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:214) ~[?:?]
   	at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:198) ~[?:?]
   	at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:190) ~[?:?]
   	at org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:282) ~[?:?]
   	at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitOperation(IcebergFilesCommitter.java:312) ~[?:?]
   	at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitDeltaTxn(IcebergFilesCommitter.java:299) ~[?:?]
   	at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitUpToCheckpoint(IcebergFilesCommitter.java:218) ~[?:?]
   	at org.apache.iceberg.flink.sink.IcebergFilesCommitter.notifyCheckpointComplete(IcebergFilesCommitter.java:188) ~[?:?]
   	at org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.notifyCheckpointComplete(StreamOperatorWrapper.java:99) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
   	at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.notifyCheckpointComplete(SubtaskCheckpointCoordinatorImpl.java:319) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
   	at org.apache.flink.streaming.runtime.tasks.StreamTask.notifyCheckpointComplete(StreamTask.java:1083) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
   	at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointCompleteAsync$11(StreamTask.java:1048) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
   	at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointOperation$13(StreamTask.java:1071) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
   	at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
   	at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
   	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:317) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
   	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:189) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
   	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:617) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
   	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:581) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
   	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:755) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
   	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:570) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
   	at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_282]
   
   
   2022-04-13 09:56:13,055 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - flink_fae_iface_ma_details_header-IcebergFilesCommitter -> Sink: flink_fae_iface_ma_details_header-IcebergSink hdfs://hdfs/user/iceberg_catalog/db__fea_fae_iceberg/flink_fae_iface_ma_details_header (1/1) (803a87e61f3dbf8a9fe0fb11dfb50357) switched from RUNNING to FAILED on 192.168.3.221:6122-088419 @ 192.168.3.221 (dataPort=42400).
   org.apache.iceberg.exceptions.NotFoundException: Failed to open input stream for file: hdfs://hdfs/user/iceberg_catalog/db__fea_fae_iceberg/flink_fae_iface_ma_details_header/metadata/snap-3945735870507280368-1-da80199a-d69d-4617-8033-824d5c01ee57.avro
   	at org.apache.iceberg.hadoop.HadoopInputFile.newStream(HadoopInputFile.java:177) ~[?:?]
   	at org.apache.iceberg.avro.AvroIterable.newFileReader(AvroIterable.java:101) ~[?:?]
   	at org.apache.iceberg.avro.AvroIterable.iterator(AvroIterable.java:77) ~[?:?]
   	at org.apache.iceberg.avro.AvroIterable.iterator(AvroIterable.java:37) ~[?:?]
   	at org.apache.iceberg.relocated.com.google.common.collect.Iterables.addAll(Iterables.java:320) ~[?:?]
   	at org.apache.iceberg.relocated.com.google.common.collect.Lists.newLinkedList(Lists.java:237) ~[?:?]
   	at org.apache.iceberg.ManifestLists.read(ManifestLists.java:46) ~[?:?]
   	at org.apache.iceberg.BaseSnapshot.cacheManifests(BaseSnapshot.java:141) ~[?:?]
   	at org.apache.iceberg.BaseSnapshot.dataManifests(BaseSnapshot.java:163) ~[?:?]
   	at org.apache.iceberg.MergingSnapshotProducer.apply(MergingSnapshotProducer.java:518) ~[?:?]
   	at org.apache.iceberg.SnapshotProducer.apply(SnapshotProducer.java:164) ~[?:?]
   	at org.apache.iceberg.SnapshotProducer.lambda$commit$2(SnapshotProducer.java:283) ~[?:?]
   	at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:404) ~[?:?]
   	at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:214) ~[?:?]
   	at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:198) ~[?:?]
   	at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:190) ~[?:?]
   	at org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:282) ~[?:?]
   	at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitOperation(IcebergFilesCommitter.java:312) ~[?:?]
   	at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitDeltaTxn(IcebergFilesCommitter.java:299) ~[?:?]
   	at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitUpToCheckpoint(IcebergFilesCommitter.java:218) ~[?:?]
   	at org.apache.iceberg.flink.sink.IcebergFilesCommitter.notifyCheckpointComplete(IcebergFilesCommitter.java:188) ~[?:?]
   	at org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.notifyCheckpointComplete(StreamOperatorWrapper.java:99) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
   	at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.notifyCheckpointComplete(SubtaskCheckpointCoordinatorImpl.java:319) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
   	at org.apache.flink.streaming.runtime.tasks.StreamTask.notifyCheckpointComplete(StreamTask.java:1083) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
   	at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointCompleteAsync$11(StreamTask.java:1048) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
   	at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointOperation$13(StreamTask.java:1071) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
   	at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
   	at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
   	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:317) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
   	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:189) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
   	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:617) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
   	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:581) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
   	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:755) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
   	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:570) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
   	at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_282]
   Caused by: java.io.FileNotFoundException: File does not exist: /user/iceberg_catalog/db__fea_fae_iceberg/flink_fae_iface_ma_details_header/metadata/snap-3945735870507280368-1-da80199a-d69d-4617-8033-824d5c01ee57.avro
   	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:85)
   	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:75)
   	at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:152)
   	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1909)
   	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:735)
   	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:415)
   	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
   	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
   	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
   	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
   	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
   	at java.security.AccessController.doPrivileged(Native Method)
   	at javax.security.auth.Subject.doAs(Subject.java:422)
   	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
   	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
   
   	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_282]
   	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_282]
   	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_282]
   	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_282]
   	at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:845) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:832) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:821) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:325) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:285) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:270) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1132) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:325) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:322) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:322) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:787) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.iceberg.hadoop.HadoopInputFile.newStream(HadoopInputFile.java:175) ~[?:?]
   	... 34 more
   Caused by: org.apache.hadoop.ipc.RemoteException: File does not exist: /user/iceberg_catalog/db__fea_fae_iceberg/flink_fae_iface_ma_details_header/metadata/snap-3945735870507280368-1-da80199a-d69d-4617-8033-824d5c01ee57.avro
   	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:85)
   	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:75)
   	at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:152)
   	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1909)
   	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:735)
   	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:415)
   	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
   	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
   	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
   	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
   	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
   	at java.security.AccessController.doPrivileged(Native Method)
   	at javax.security.auth.Subject.doAs(Subject.java:422)
   	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
   	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
   
   	at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1489) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.hadoop.ipc.Client.call(Client.java:1435) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.hadoop.ipc.Client.call(Client.java:1345) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at com.sun.proxy.$Proxy45.getBlockLocations(Unknown Source) ~[?:?]
   	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:259) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at sun.reflect.GeneratedMethodAccessor118.invoke(Unknown Source) ~[?:?]
   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_282]
   	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_282]
   	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:346) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at com.sun.proxy.$Proxy46.getBlockLocations(Unknown Source) ~[?:?]
   	at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:843) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:832) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:821) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:325) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:285) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:270) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1132) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:325) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:322) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:322) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:787) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
   	at org.apache.iceberg.hadoop.HadoopInputFile.newStream(HadoopInputFile.java:175) ~[?:?]
   	... 34 more
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] the snapshot file is lost when write iceberg using flink Failed to open input stream for file File does not exist [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra closed issue #4550: the snapshot file is lost when write iceberg using flink   Failed to open input stream for file    File does not exist
URL: https://github.com/apache/iceberg/issues/4550


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] owencwl commented on issue #4550: the snapshot file is lost when write iceberg using flink Failed to open input stream for file File does not exist

Posted by GitBox <gi...@apache.org>.
owencwl commented on issue #4550:
URL: https://github.com/apache/iceberg/issues/4550#issuecomment-1223479872

   内部有重试机制,以下文件有可能存在(某一个):
   snap-3945735870507280368-2-da80199a-d69d-4617-8033-824d5c01ee57.avro
   snap-3945735870507280368-3-da80199a-d69d-4617-8033-824d5c01ee57.avro
   
   我也出现过此问题:
   当hive-metastore或者内部的mysql,重启、挂掉、宕机时,然后继续向iceberg写数据时,就会出现snap-xxxx-x-xxx.avro找不到文件,可能iceberg在生成minifastList文件和commit时存在bug
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] zstraw commented on issue #4550: the snapshot file is lost when write iceberg using flink Failed to open input stream for file File does not exist

Posted by GitBox <gi...@apache.org>.
zstraw commented on issue #4550:
URL: https://github.com/apache/iceberg/issues/4550#issuecomment-1340655807

   > I have solved this problem. Thank you. My problem mainly occurs when InMemoryLockManager releases the heartbeat of the lock and reports a NullPointerException; I rewrote InMemoryLockManager to solve this problem. 在 2022年12月6日 ***@***.***> 写道: After deeping into iceberg code and the log, I can reproduce it in debugging locally. The scenario may happens in the process of Flink cancelling. IcebergFileCommitter is going to commit file. In the step of rename metadata.json(org.apache.iceberg.hadoop.HadoopTableOperations#renameToFinal), org.apache.hadoop.ipc.Client.call encounters InterruptedIOException. I suspect it comes from Flink task cancelling. On the other hand, Hdfs has renamed the metada.json file sucessfully. After rename fails, it's supposed to retry. But the thread encounters InterruptedException in sleeping(org.apache.iceberg.util.Tasks#runTaskWithRetry). Then it will throw a RuntimeException. And the version-hint will not be updated. The RuntimeException lea
 ds to rollback in org.apache.iceberg.BaseTransaction(#cleanUpOnCommitFailure), which will delete manifest list (snap-XXX). — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>
   
   But I think this problem is still a bug in Iceberg's commit procedure. There are several tasks encountering snap-XXX.avro files lost in our env.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] wxjzlm commented on issue #4550: the snapshot file is lost when write iceberg using flink Failed to open input stream for file File does not exist

Posted by GitBox <gi...@apache.org>.
wxjzlm commented on issue #4550:
URL: https://github.com/apache/iceberg/issues/4550#issuecomment-1323337067

   > 
   
   频繁因为这个问题,flink作业失败


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] wxjzlm commented on issue #4550: the snapshot file is lost when write iceberg using flink Failed to open input stream for file File does not exist

Posted by GitBox <gi...@apache.org>.
wxjzlm commented on issue #4550:
URL: https://github.com/apache/iceberg/issues/4550#issuecomment-1339282266

   I have solved this problem. Thank you. My problem mainly occurs when InMemoryLockManager releases the heartbeat of the lock and reports a NullPointerException; I rewrote InMemoryLockManager to solve this problem.
   
   
   在 2022年12月6日 ***@***.***> 写道:
   
   
   After deeping into iceberg code and the log, I can reproduce it in debugging locally.
   The scenario may happens in the process of Flink cancelling.
   IcebergFileCommitter is going to commit file. In the step of rename metadata.json(org.apache.iceberg.hadoop.HadoopTableOperations#renameToFinal), org.apache.hadoop.ipc.Client.call encounters InterruptedIOException. I suspect it comes from Flink task cancelling. On the other hand, Hdfs has renamed the metada.json file sucessfully. After rename fails, it's supposed to retry. But the thread encounters InterruptedException in sleeping(org.apache.iceberg.util.Tasks#runTaskWithRetry). Then it will throw a RuntimeException. And the version-hint will not be updated. The RuntimeException leads to rollback in org.apache.iceberg.BaseTransaction(#cleanUpOnCommitFailure), which will delete manifest list (snap-XXX).
   —
   Reply to this email directly, view it on GitHub, or unsubscribe.
   You are receiving this because you commented.Message ID: ***@***.***>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] zstraw commented on issue #4550: the snapshot file is lost when write iceberg using flink Failed to open input stream for file File does not exist

Posted by GitBox <gi...@apache.org>.
zstraw commented on issue #4550:
URL: https://github.com/apache/iceberg/issues/4550#issuecomment-1338908066

   After deeping into iceberg code and the log, I can reproduce it in debugging locally.
   
   The scenario may happens in the process of Flink cancelling.
   1. IcebergFileCommitter is going to commit file. In the step of **rename** metadata.json(org.apache.iceberg.hadoop.HadoopTableOperations#renameToFinal), org.apache.hadoop.ipc.Client.call encounters **InterruptedIOException**. I suspect it comes from Flink task cancelling. On the other hand, **Hdfs has renamed the metada.json file sucessfully**.
   2. After rename fails, it's supposed to retry. But the thread encounters InterruptedException in sleeping(org.apache.iceberg.util.Tasks#runTaskWithRetry). Then it will throw a RuntimeException. And the version-hint will not be updated.
   3. The RuntimeException leads to **rollback** in org.apache.iceberg.BaseTransaction(#cleanUpOnCommitFailure), which will delete manifest list (snap-XXX).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] 245831311 commented on issue #4550: the snapshot file is lost when write iceberg using flink Failed to open input stream for file File does not exist

Posted by GitBox <gi...@apache.org>.
245831311 commented on issue #4550:
URL: https://github.com/apache/iceberg/issues/4550#issuecomment-1340578855

   > I have solved this problem. Thank you. My problem mainly occurs when InMemoryLockManager releases the heartbeat of the lock and reports a NullPointerException; I rewrote InMemoryLockManager to solve this problem. 在 2022年12月6日 ***@***.***> 写道: After deeping into iceberg code and the log, I can reproduce it in debugging locally. The scenario may happens in the process of Flink cancelling. IcebergFileCommitter is going to commit file. In the step of rename metadata.json(org.apache.iceberg.hadoop.HadoopTableOperations#renameToFinal), org.apache.hadoop.ipc.Client.call encounters InterruptedIOException. I suspect it comes from Flink task cancelling. On the other hand, Hdfs has renamed the metada.json file sucessfully. After rename fails, it's supposed to retry. But the thread encounters InterruptedException in sleeping(org.apache.iceberg.util.Tasks#runTaskWithRetry). Then it will throw a RuntimeException. And the version-hint will not be updated. The RuntimeException lea
 ds to rollback in org.apache.iceberg.BaseTransaction(#cleanUpOnCommitFailure), which will delete manifest list (snap-XXX). — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>
   
   能否更具体描述下吗?我被这个问题困扰了挺久 ,感谢


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] 245831311 commented on issue #4550: the snapshot file is lost when write iceberg using flink Failed to open input stream for file File does not exist

Posted by GitBox <gi...@apache.org>.
245831311 commented on issue #4550:
URL: https://github.com/apache/iceberg/issues/4550#issuecomment-1298869839

   > 内部有重试机制,以下文件有可能存在(某一个): snap-3945735870507280368-2-da80199a-d69d-4617-8033-824d5c01ee57.avro snap-3945735870507280368-3-da80199a-d69d-4617-8033-824d5c01ee57.avro
   > 
   > 我也出现过此问题: 当hive-metastore或者内部的mysql,重启、挂掉、宕机时,然后继续向iceberg写数据时,就会出现snap-xxxx-x-xxx.avro找不到文件,可能iceberg在生成minifastList文件和commit时存在bug
   
   有什么方法解决这个问题,我也出现了


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] spancer commented on issue #4550: the snapshot file is lost when write iceberg using flink Failed to open input stream for file File does not exist

Posted by GitBox <gi...@apache.org>.
spancer commented on issue #4550:
URL: https://github.com/apache/iceberg/issues/4550#issuecomment-1225307952

   anyway to reproduce this issue?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] tiandiyouxue commented on issue #4550: the snapshot file is lost when write iceberg using flink Failed to open input stream for file File does not exist

Posted by "tiandiyouxue (via GitHub)" <gi...@apache.org>.
tiandiyouxue commented on issue #4550:
URL: https://github.com/apache/iceberg/issues/4550#issuecomment-1606507038

   > I have solved this problem. Thank you. My problem mainly occurs when InMemoryLockManager releases the heartbeat of the lock and reports a NullPointerException; I rewrote InMemoryLockManager to solve this problem. 在 2022年12月6日 ***@***.***> 写道: After deeping into iceberg code and the log, I can reproduce it in debugging locally. The scenario may happens in the process of Flink cancelling. IcebergFileCommitter is going to commit file. In the step of rename metadata.json(org.apache.iceberg.hadoop.HadoopTableOperations#renameToFinal), org.apache.hadoop.ipc.Client.call encounters InterruptedIOException. I suspect it comes from Flink task cancelling. On the other hand, Hdfs has renamed the metada.json file sucessfully. After rename fails, it's supposed to retry. But the thread encounters InterruptedException in sleeping(org.apache.iceberg.util.Tasks#runTaskWithRetry). Then it will throw a RuntimeException. And the version-hint will not be updated. The RuntimeException lea
 ds to rollback in org.apache.iceberg.BaseTransaction(#cleanUpOnCommitFailure), which will delete manifest list (snap-XXX). — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] jxwnhj0717 commented on issue #4550: the snapshot file is lost when write iceberg using flink Failed to open input stream for file File does not exist

Posted by GitBox <gi...@apache.org>.
jxwnhj0717 commented on issue #4550:
URL: https://github.com/apache/iceberg/issues/4550#issuecomment-1381352493

   I added logs for locking and releasing locks in LockManagers$InMemoryLockManager, reproducing the bad table issue:
   `
   [IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_base_city (1/1)#2] 2023-01-11 18:46:03,672 INFO  org.apache.iceberg.util.LockManagers$InMemoryLockManager     [] - acquiring lock. entityId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_base_city/metadata/v749.metadata.json ownerId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_base_city/metadata/27a6b56d-7747-4a7b-9da1-694c5c6f8b3b.metadata.json
   [IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_rep_person (1/1)#2] 2023-01-11 18:46:03,672 INFO  org.apache.iceberg.util.LockManagers$InMemoryLockManager     [] - acquiring lock. entityId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_rep_person/metadata/v756.metadata.json ownerId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_rep_person/metadata/db1c3994-dd04-410a-aa95-6516b7615246.metadata.json
   [IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_rep_person (1/1)#2] 2023-01-11 18:46:03,672 INFO  org.apache.iceberg.util.LockManagers$InMemoryLockManager     [] - acquired lock. entityId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_rep_person/metadata/v756.metadata.json ownerId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_rep_person/metadata/db1c3994-dd04-410a-aa95-6516b7615246.metadata.json
   [IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_base_city (1/1)#2] 2023-01-11 18:46:03,672 INFO  org.apache.iceberg.util.LockManagers$InMemoryLockManager     [] - acquired lock. entityId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_base_city/metadata/v749.metadata.json ownerId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_base_city/metadata/27a6b56d-7747-4a7b-9da1-694c5c6f8b3b.metadata.json
   [IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_base_city (1/1)#2] 2023-01-11 18:46:03,672 INFO  org.apache.iceberg.util.LockManagers$InMemoryLockManager     [] - add lock heartbeat. entityId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_base_city/metadata/v749.metadata.json ownerId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_base_city/metadata/27a6b56d-7747-4a7b-9da1-694c5c6f8b3b.metadata.json
   [IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_rep_person (1/1)#2] 2023-01-11 18:46:03,672 INFO  org.apache.iceberg.util.LockManagers$InMemoryLockManager     [] - add lock heartbeat. entityId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_rep_person/metadata/v756.metadata.json ownerId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_rep_person/metadata/db1c3994-dd04-410a-aa95-6516b7615246.metadata.json
   [IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_rep_docsource_type (1/1)#2] 2023-01-11 18:46:03,674 INFO  org.apache.iceberg.util.LockManagers$InMemoryLockManager     [] - releasing lock. entityId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_rep_docsource_type/metadata/v749.metadata.json ownerId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_rep_docsource_type/metadata/22f4f9c5-35d2-467c-8263-d26fd6548017.metadata.json
   [IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_rep_docsource_type (1/1)#2] 2023-01-11 18:46:03,674 INFO  org.apache.iceberg.util.LockManagers$InMemoryLockManager     [] - released lock. entityId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_rep_docsource_type/metadata/v749.metadata.json ownerId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_rep_docsource_type/metadata/22f4f9c5-35d2-467c-8263-d26fd6548017.metadata.json
   [IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_rep_docsource_type (1/1)#2] 2023-01-11 18:46:03,674 INFO  org.apache.iceberg.hadoop.HadoopTableOperations              [] - Committed a new metadata file hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_rep_docsource_type/metadata/v749.metadata.json
   [IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_base_province (1/1)#2] 2023-01-11 18:46:03,677 INFO  org.apache.iceberg.util.LockManagers$InMemoryLockManager     [] - releasing lock. entityId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_base_province/metadata/v749.metadata.json ownerId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_base_province/metadata/ced05720-8f32-49c9-884d-49b604290622.metadata.json
   [IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_base_province (1/1)#2] 2023-01-11 18:46:03,677 INFO  org.apache.iceberg.util.LockManagers$InMemoryLockManager     [] - released lock. entityId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_base_province/metadata/v749.metadata.json ownerId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_base_province/metadata/ced05720-8f32-49c9-884d-49b604290622.metadata.json
   [IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_base_province (1/1)#2] 2023-01-11 18:46:03,677 INFO  org.apache.iceberg.hadoop.HadoopTableOperations              [] - Committed a new metadata file hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_base_province/metadata/v749.metadata.json
   [IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_base_city (1/1)#2] 2023-01-11 18:46:03,679 INFO  org.apache.iceberg.util.LockManagers$InMemoryLockManager     [] - releasing lock. entityId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_base_city/metadata/v749.metadata.json ownerId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_base_city/metadata/27a6b56d-7747-4a7b-9da1-694c5c6f8b3b.metadata.json
   [IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_rep_person (1/1)#2] 2023-01-11 18:46:03,679 INFO  org.apache.iceberg.util.LockManagers$InMemoryLockManager     [] - releasing lock. entityId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_rep_person/metadata/v756.metadata.json ownerId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_rep_person/metadata/db1c3994-dd04-410a-aa95-6516b7615246.metadata.json
   [IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_rep_person (1/1)#2] 2023-01-11 18:46:03,679 INFO  org.apache.iceberg.util.LockManagers$InMemoryLockManager     [] - released lock. entityId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_rep_person/metadata/v756.metadata.json ownerId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_rep_person/metadata/db1c3994-dd04-410a-aa95-6516b7615246.metadata.json
   [IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_rep_person (1/1)#2] 2023-01-11 18:46:03,679 INFO  org.apache.iceberg.hadoop.HadoopTableOperations              [] - Committed a new metadata file hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_rep_person/metadata/v756.metadata.json
   [IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_base_city (1/1)#2] 2023-01-11 18:46:03,684 WARN  org.apache.flink.runtime.taskmanager.Task                    [] - IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_base_city (1/1)#2 (5dec3e28251735766bb3eb423ca5a45c) switched from RUNNING to FAILED with failure cause: java.lang.NullPointerException
   	at org.apache.iceberg.util.LockManagers$InMemoryLockManager.release(LockManagers.java:235)
   	at org.apache.iceberg.hadoop.HadoopTableOperations.renameToFinal(HadoopTableOperations.java:377)
   	at org.apache.iceberg.hadoop.HadoopTableOperations.commit(HadoopTableOperations.java:159)
   	at org.apache.iceberg.SnapshotProducer.lambda$commit$2(SnapshotProducer.java:317)
   	at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:404)
   	at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:214)
   	at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:198)
   	at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:190)
   	at org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:295)
   	at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitOperation(IcebergFilesCommitter.java:312)
   	at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitDeltaTxn(IcebergFilesCommitter.java:276)
   	at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitUpToCheckpoint(IcebergFilesCommitter.java:218)
   	at org.apache.iceberg.flink.sink.IcebergFilesCommitter.notifyCheckpointComplete(IcebergFilesCommitter.java:188)
   	at org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.notifyCheckpointComplete(StreamOperatorWrapper.java:99)
   	at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.notifyCheckpointComplete(SubtaskCheckpointCoordinatorImpl.java:334)
   	at org.apache.flink.streaming.runtime.tasks.StreamTask.notifyCheckpointComplete(StreamTask.java:1171)
   	at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointCompleteAsync$10(StreamTask.java:1136)
   	at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointOperation$12(StreamTask.java:1159)
   	at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
   	at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
   	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:344)
   	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:330)
   	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:202)
   	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:684)
   	at org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:639)
   	at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650)
   	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:623)
   	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779)
   	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
   	at java.lang.Thread.run(Thread.java:748)
   `
   
   The code is as follows:
   `
       @VisibleForTesting
       void acquireOnce(String entityId, String ownerId) {
         InMemoryLockContent content = LOCKS.get(entityId);
         if (content != null && content.expireMs() > System.currentTimeMillis()) {
           throw new IllegalStateException(String.format("Lock for %s currently held by %s, expiration: %s",
                   entityId, content.ownerId(), content.expireMs()));
         }
   
         LOG.info("acquiring lock. entityId:{} ownerId:{}", entityId, ownerId);
         long expiration = System.currentTimeMillis() + heartbeatTimeoutMs();
         boolean succeed;
         if (content == null) {
           InMemoryLockContent previous = LOCKS.putIfAbsent(
                   entityId, new InMemoryLockContent(ownerId, expiration));
           succeed = previous == null;
         } else {
           succeed = LOCKS.replace(entityId, content, new InMemoryLockContent(ownerId, expiration));
         }
   
         if (succeed) {
           LOG.info("acquired lock. entityId:{} ownerId:{}", entityId, ownerId);
           // cleanup old heartbeat
           if (HEARTBEATS.containsKey(entityId)) {
             HEARTBEATS.remove(entityId).cancel(false);
             LOG.info("remove lock heartbeat. entityId:{} ownerId:{}", entityId, ownerId);
           }
   
           HEARTBEATS.put(entityId, scheduler().scheduleAtFixedRate(() -> {
             InMemoryLockContent lastContent = LOCKS.get(entityId);
             try {
               long newExpiration = System.currentTimeMillis() + heartbeatTimeoutMs();
               LOCKS.replace(entityId, lastContent, new InMemoryLockContent(ownerId, newExpiration));
             } catch (NullPointerException e) {
               throw new RuntimeException("Cannot heartbeat to a deleted lock " + entityId, e);
             }
   
           }, 0, heartbeatIntervalMs(), TimeUnit.MILLISECONDS));
           LOG.info("add lock heartbeat. entityId:{} ownerId:{}", entityId, ownerId);
   
         } else {
           throw new IllegalStateException("Unable to acquire lock " + entityId);
         }
       }
       @Override
       public boolean release(String entityId, String ownerId) {
         LOG.info("releasing lock. entityId:{} ownerId:{}", entityId, ownerId);
         InMemoryLockContent currentContent = LOCKS.get(entityId);
         if (currentContent == null) {
           LOG.error("Cannot find lock for entity {}", entityId);
           return false;
         }
   
         if (!currentContent.ownerId().equals(ownerId)) {
           LOG.error("Cannot unlock {} by {}, current owner: {}", entityId, ownerId, currentContent.ownerId());
           return false;
         }
   
         HEARTBEATS.remove(entityId).cancel(false);
         LOCKS.remove(entityId);
         LOG.info("released lock. entityId:{} ownerId:{}", entityId, ownerId);
         return true;
       }
   `
   
   HEARTBEATS.remove(entityId) is empty when the lock is released, strangely, 2023-01-11 18:46:03, 672 added entityId heartbeart, 2023-01-11 18:46:03, 684 can not find the corresponding heartbeat, the whole process is single-threaded.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org