You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/04/13 09:27:57 UTC
[GitHub] [iceberg] wndemon opened a new issue, #4550: the snapshot file is lost when write iceberg using flink Failed to open input stream for file File does not exist
wndemon opened a new issue, #4550:
URL: https://github.com/apache/iceberg/issues/4550
通过iceberg-flink-runtime-1.12-0.13.1.jar 使用flink写iceberg,偶尔会出现下述问题,快照文件丢失,找不到快照文件
write iceberg from flink-runtime-1.12-0.13.1.jar, the following problems occasionally occur: the snapshot file is lost and cannot be found
2022-04-13 09:54:48,657 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - flink_fae_iface_ma_details_header-IcebergFilesCommitter -> Sink: flink_fae_iface_ma_details_header-IcebergSink hdfs://hdfs/user/iceberg_catalog/db__fea_fae_iceberg/flink_fae_iface_ma_details_header (1/1) (09b19ba94ff66428538bb4bb3dae372b) switched from RUNNING to FAILED on 192.168.3.221:6122-088419 @ 192.168.3.221 (dataPort=42400).
java.lang.NullPointerException: null
at org.apache.iceberg.util.LockManagers$InMemoryLockManager.release(LockManagers.java:225) ~[?:?]
at org.apache.iceberg.hadoop.HadoopTableOperations.renameToFinal(HadoopTableOperations.java:374) ~[?:?]
at org.apache.iceberg.hadoop.HadoopTableOperations.commit(HadoopTableOperations.java:158) ~[?:?]
at org.apache.iceberg.SnapshotProducer.lambda$commit$2(SnapshotProducer.java:300) ~[?:?]
at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:404) ~[?:?]
at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:214) ~[?:?]
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:198) ~[?:?]
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:190) ~[?:?]
at org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:282) ~[?:?]
at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitOperation(IcebergFilesCommitter.java:312) ~[?:?]
at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitDeltaTxn(IcebergFilesCommitter.java:299) ~[?:?]
at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitUpToCheckpoint(IcebergFilesCommitter.java:218) ~[?:?]
at org.apache.iceberg.flink.sink.IcebergFilesCommitter.notifyCheckpointComplete(IcebergFilesCommitter.java:188) ~[?:?]
at org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.notifyCheckpointComplete(StreamOperatorWrapper.java:99) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.notifyCheckpointComplete(SubtaskCheckpointCoordinatorImpl.java:319) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
at org.apache.flink.streaming.runtime.tasks.StreamTask.notifyCheckpointComplete(StreamTask.java:1083) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointCompleteAsync$11(StreamTask.java:1048) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointOperation$13(StreamTask.java:1071) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:317) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:189) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:617) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:581) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:755) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:570) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_282]
2022-04-13 09:56:13,055 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - flink_fae_iface_ma_details_header-IcebergFilesCommitter -> Sink: flink_fae_iface_ma_details_header-IcebergSink hdfs://hdfs/user/iceberg_catalog/db__fea_fae_iceberg/flink_fae_iface_ma_details_header (1/1) (803a87e61f3dbf8a9fe0fb11dfb50357) switched from RUNNING to FAILED on 192.168.3.221:6122-088419 @ 192.168.3.221 (dataPort=42400).
org.apache.iceberg.exceptions.NotFoundException: Failed to open input stream for file: hdfs://hdfs/user/iceberg_catalog/db__fea_fae_iceberg/flink_fae_iface_ma_details_header/metadata/snap-3945735870507280368-1-da80199a-d69d-4617-8033-824d5c01ee57.avro
at org.apache.iceberg.hadoop.HadoopInputFile.newStream(HadoopInputFile.java:177) ~[?:?]
at org.apache.iceberg.avro.AvroIterable.newFileReader(AvroIterable.java:101) ~[?:?]
at org.apache.iceberg.avro.AvroIterable.iterator(AvroIterable.java:77) ~[?:?]
at org.apache.iceberg.avro.AvroIterable.iterator(AvroIterable.java:37) ~[?:?]
at org.apache.iceberg.relocated.com.google.common.collect.Iterables.addAll(Iterables.java:320) ~[?:?]
at org.apache.iceberg.relocated.com.google.common.collect.Lists.newLinkedList(Lists.java:237) ~[?:?]
at org.apache.iceberg.ManifestLists.read(ManifestLists.java:46) ~[?:?]
at org.apache.iceberg.BaseSnapshot.cacheManifests(BaseSnapshot.java:141) ~[?:?]
at org.apache.iceberg.BaseSnapshot.dataManifests(BaseSnapshot.java:163) ~[?:?]
at org.apache.iceberg.MergingSnapshotProducer.apply(MergingSnapshotProducer.java:518) ~[?:?]
at org.apache.iceberg.SnapshotProducer.apply(SnapshotProducer.java:164) ~[?:?]
at org.apache.iceberg.SnapshotProducer.lambda$commit$2(SnapshotProducer.java:283) ~[?:?]
at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:404) ~[?:?]
at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:214) ~[?:?]
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:198) ~[?:?]
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:190) ~[?:?]
at org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:282) ~[?:?]
at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitOperation(IcebergFilesCommitter.java:312) ~[?:?]
at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitDeltaTxn(IcebergFilesCommitter.java:299) ~[?:?]
at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitUpToCheckpoint(IcebergFilesCommitter.java:218) ~[?:?]
at org.apache.iceberg.flink.sink.IcebergFilesCommitter.notifyCheckpointComplete(IcebergFilesCommitter.java:188) ~[?:?]
at org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.notifyCheckpointComplete(StreamOperatorWrapper.java:99) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.notifyCheckpointComplete(SubtaskCheckpointCoordinatorImpl.java:319) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
at org.apache.flink.streaming.runtime.tasks.StreamTask.notifyCheckpointComplete(StreamTask.java:1083) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointCompleteAsync$11(StreamTask.java:1048) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointOperation$13(StreamTask.java:1071) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:317) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:189) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:617) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:581) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:755) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:570) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_282]
Caused by: java.io.FileNotFoundException: File does not exist: /user/iceberg_catalog/db__fea_fae_iceberg/flink_fae_iface_ma_details_header/metadata/snap-3945735870507280368-1-da80199a-d69d-4617-8033-824d5c01ee57.avro
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:85)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:75)
at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:152)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1909)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:735)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:415)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_282]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_282]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_282]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_282]
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:845) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:832) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:821) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:325) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:285) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:270) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1132) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:325) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:322) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:322) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:787) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.iceberg.hadoop.HadoopInputFile.newStream(HadoopInputFile.java:175) ~[?:?]
... 34 more
Caused by: org.apache.hadoop.ipc.RemoteException: File does not exist: /user/iceberg_catalog/db__fea_fae_iceberg/flink_fae_iface_ma_details_header/metadata/snap-3945735870507280368-1-da80199a-d69d-4617-8033-824d5c01ee57.avro
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:85)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:75)
at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:152)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1909)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:735)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:415)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1489) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.hadoop.ipc.Client.call(Client.java:1435) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.hadoop.ipc.Client.call(Client.java:1345) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at com.sun.proxy.$Proxy45.getBlockLocations(Unknown Source) ~[?:?]
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:259) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at sun.reflect.GeneratedMethodAccessor118.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_282]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_282]
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:346) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at com.sun.proxy.$Proxy46.getBlockLocations(Unknown Source) ~[?:?]
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:843) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:832) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:821) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:325) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:285) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:270) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1132) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:325) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:322) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:322) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:787) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at org.apache.iceberg.hadoop.HadoopInputFile.newStream(HadoopInputFile.java:175) ~[?:?]
... 34 more
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] the snapshot file is lost when write iceberg using flink Failed to open input stream for file File does not exist [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra closed issue #4550: the snapshot file is lost when write iceberg using flink Failed to open input stream for file File does not exist
URL: https://github.com/apache/iceberg/issues/4550
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] owencwl commented on issue #4550: the snapshot file is lost when write iceberg using flink Failed to open input stream for file File does not exist
Posted by GitBox <gi...@apache.org>.
owencwl commented on issue #4550:
URL: https://github.com/apache/iceberg/issues/4550#issuecomment-1223479872
内部有重试机制,以下文件有可能存在(某一个):
snap-3945735870507280368-2-da80199a-d69d-4617-8033-824d5c01ee57.avro
snap-3945735870507280368-3-da80199a-d69d-4617-8033-824d5c01ee57.avro
我也出现过此问题:
当hive-metastore或者内部的mysql,重启、挂掉、宕机时,然后继续向iceberg写数据时,就会出现snap-xxxx-x-xxx.avro找不到文件,可能iceberg在生成minifastList文件和commit时存在bug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] zstraw commented on issue #4550: the snapshot file is lost when write iceberg using flink Failed to open input stream for file File does not exist
Posted by GitBox <gi...@apache.org>.
zstraw commented on issue #4550:
URL: https://github.com/apache/iceberg/issues/4550#issuecomment-1340655807
> I have solved this problem. Thank you. My problem mainly occurs when InMemoryLockManager releases the heartbeat of the lock and reports a NullPointerException; I rewrote InMemoryLockManager to solve this problem. 在 2022年12月6日 ***@***.***> 写道: After deeping into iceberg code and the log, I can reproduce it in debugging locally. The scenario may happens in the process of Flink cancelling. IcebergFileCommitter is going to commit file. In the step of rename metadata.json(org.apache.iceberg.hadoop.HadoopTableOperations#renameToFinal), org.apache.hadoop.ipc.Client.call encounters InterruptedIOException. I suspect it comes from Flink task cancelling. On the other hand, Hdfs has renamed the metada.json file sucessfully. After rename fails, it's supposed to retry. But the thread encounters InterruptedException in sleeping(org.apache.iceberg.util.Tasks#runTaskWithRetry). Then it will throw a RuntimeException. And the version-hint will not be updated. The RuntimeException lea
ds to rollback in org.apache.iceberg.BaseTransaction(#cleanUpOnCommitFailure), which will delete manifest list (snap-XXX). — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>
But I think this problem is still a bug in Iceberg's commit procedure. There are several tasks encountering snap-XXX.avro files lost in our env.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] wxjzlm commented on issue #4550: the snapshot file is lost when write iceberg using flink Failed to open input stream for file File does not exist
Posted by GitBox <gi...@apache.org>.
wxjzlm commented on issue #4550:
URL: https://github.com/apache/iceberg/issues/4550#issuecomment-1323337067
>
频繁因为这个问题,flink作业失败
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] wxjzlm commented on issue #4550: the snapshot file is lost when write iceberg using flink Failed to open input stream for file File does not exist
Posted by GitBox <gi...@apache.org>.
wxjzlm commented on issue #4550:
URL: https://github.com/apache/iceberg/issues/4550#issuecomment-1339282266
I have solved this problem. Thank you. My problem mainly occurs when InMemoryLockManager releases the heartbeat of the lock and reports a NullPointerException; I rewrote InMemoryLockManager to solve this problem.
在 2022年12月6日 ***@***.***> 写道:
After deeping into iceberg code and the log, I can reproduce it in debugging locally.
The scenario may happens in the process of Flink cancelling.
IcebergFileCommitter is going to commit file. In the step of rename metadata.json(org.apache.iceberg.hadoop.HadoopTableOperations#renameToFinal), org.apache.hadoop.ipc.Client.call encounters InterruptedIOException. I suspect it comes from Flink task cancelling. On the other hand, Hdfs has renamed the metada.json file sucessfully. After rename fails, it's supposed to retry. But the thread encounters InterruptedException in sleeping(org.apache.iceberg.util.Tasks#runTaskWithRetry). Then it will throw a RuntimeException. And the version-hint will not be updated. The RuntimeException leads to rollback in org.apache.iceberg.BaseTransaction(#cleanUpOnCommitFailure), which will delete manifest list (snap-XXX).
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.Message ID: ***@***.***>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] zstraw commented on issue #4550: the snapshot file is lost when write iceberg using flink Failed to open input stream for file File does not exist
Posted by GitBox <gi...@apache.org>.
zstraw commented on issue #4550:
URL: https://github.com/apache/iceberg/issues/4550#issuecomment-1338908066
After deeping into iceberg code and the log, I can reproduce it in debugging locally.
The scenario may happens in the process of Flink cancelling.
1. IcebergFileCommitter is going to commit file. In the step of **rename** metadata.json(org.apache.iceberg.hadoop.HadoopTableOperations#renameToFinal), org.apache.hadoop.ipc.Client.call encounters **InterruptedIOException**. I suspect it comes from Flink task cancelling. On the other hand, **Hdfs has renamed the metada.json file sucessfully**.
2. After rename fails, it's supposed to retry. But the thread encounters InterruptedException in sleeping(org.apache.iceberg.util.Tasks#runTaskWithRetry). Then it will throw a RuntimeException. And the version-hint will not be updated.
3. The RuntimeException leads to **rollback** in org.apache.iceberg.BaseTransaction(#cleanUpOnCommitFailure), which will delete manifest list (snap-XXX).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] 245831311 commented on issue #4550: the snapshot file is lost when write iceberg using flink Failed to open input stream for file File does not exist
Posted by GitBox <gi...@apache.org>.
245831311 commented on issue #4550:
URL: https://github.com/apache/iceberg/issues/4550#issuecomment-1340578855
> I have solved this problem. Thank you. My problem mainly occurs when InMemoryLockManager releases the heartbeat of the lock and reports a NullPointerException; I rewrote InMemoryLockManager to solve this problem. 在 2022年12月6日 ***@***.***> 写道: After deeping into iceberg code and the log, I can reproduce it in debugging locally. The scenario may happens in the process of Flink cancelling. IcebergFileCommitter is going to commit file. In the step of rename metadata.json(org.apache.iceberg.hadoop.HadoopTableOperations#renameToFinal), org.apache.hadoop.ipc.Client.call encounters InterruptedIOException. I suspect it comes from Flink task cancelling. On the other hand, Hdfs has renamed the metada.json file sucessfully. After rename fails, it's supposed to retry. But the thread encounters InterruptedException in sleeping(org.apache.iceberg.util.Tasks#runTaskWithRetry). Then it will throw a RuntimeException. And the version-hint will not be updated. The RuntimeException lea
ds to rollback in org.apache.iceberg.BaseTransaction(#cleanUpOnCommitFailure), which will delete manifest list (snap-XXX). — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>
能否更具体描述下吗?我被这个问题困扰了挺久 ,感谢
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] 245831311 commented on issue #4550: the snapshot file is lost when write iceberg using flink Failed to open input stream for file File does not exist
Posted by GitBox <gi...@apache.org>.
245831311 commented on issue #4550:
URL: https://github.com/apache/iceberg/issues/4550#issuecomment-1298869839
> 内部有重试机制,以下文件有可能存在(某一个): snap-3945735870507280368-2-da80199a-d69d-4617-8033-824d5c01ee57.avro snap-3945735870507280368-3-da80199a-d69d-4617-8033-824d5c01ee57.avro
>
> 我也出现过此问题: 当hive-metastore或者内部的mysql,重启、挂掉、宕机时,然后继续向iceberg写数据时,就会出现snap-xxxx-x-xxx.avro找不到文件,可能iceberg在生成minifastList文件和commit时存在bug
有什么方法解决这个问题,我也出现了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] spancer commented on issue #4550: the snapshot file is lost when write iceberg using flink Failed to open input stream for file File does not exist
Posted by GitBox <gi...@apache.org>.
spancer commented on issue #4550:
URL: https://github.com/apache/iceberg/issues/4550#issuecomment-1225307952
anyway to reproduce this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] tiandiyouxue commented on issue #4550: the snapshot file is lost when write iceberg using flink Failed to open input stream for file File does not exist
Posted by "tiandiyouxue (via GitHub)" <gi...@apache.org>.
tiandiyouxue commented on issue #4550:
URL: https://github.com/apache/iceberg/issues/4550#issuecomment-1606507038
> I have solved this problem. Thank you. My problem mainly occurs when InMemoryLockManager releases the heartbeat of the lock and reports a NullPointerException; I rewrote InMemoryLockManager to solve this problem. 在 2022年12月6日 ***@***.***> 写道: After deeping into iceberg code and the log, I can reproduce it in debugging locally. The scenario may happens in the process of Flink cancelling. IcebergFileCommitter is going to commit file. In the step of rename metadata.json(org.apache.iceberg.hadoop.HadoopTableOperations#renameToFinal), org.apache.hadoop.ipc.Client.call encounters InterruptedIOException. I suspect it comes from Flink task cancelling. On the other hand, Hdfs has renamed the metada.json file sucessfully. After rename fails, it's supposed to retry. But the thread encounters InterruptedException in sleeping(org.apache.iceberg.util.Tasks#runTaskWithRetry). Then it will throw a RuntimeException. And the version-hint will not be updated. The RuntimeException lea
ds to rollback in org.apache.iceberg.BaseTransaction(#cleanUpOnCommitFailure), which will delete manifest list (snap-XXX). — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jxwnhj0717 commented on issue #4550: the snapshot file is lost when write iceberg using flink Failed to open input stream for file File does not exist
Posted by GitBox <gi...@apache.org>.
jxwnhj0717 commented on issue #4550:
URL: https://github.com/apache/iceberg/issues/4550#issuecomment-1381352493
I added logs for locking and releasing locks in LockManagers$InMemoryLockManager, reproducing the bad table issue:
`
[IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_base_city (1/1)#2] 2023-01-11 18:46:03,672 INFO org.apache.iceberg.util.LockManagers$InMemoryLockManager [] - acquiring lock. entityId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_base_city/metadata/v749.metadata.json ownerId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_base_city/metadata/27a6b56d-7747-4a7b-9da1-694c5c6f8b3b.metadata.json
[IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_rep_person (1/1)#2] 2023-01-11 18:46:03,672 INFO org.apache.iceberg.util.LockManagers$InMemoryLockManager [] - acquiring lock. entityId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_rep_person/metadata/v756.metadata.json ownerId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_rep_person/metadata/db1c3994-dd04-410a-aa95-6516b7615246.metadata.json
[IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_rep_person (1/1)#2] 2023-01-11 18:46:03,672 INFO org.apache.iceberg.util.LockManagers$InMemoryLockManager [] - acquired lock. entityId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_rep_person/metadata/v756.metadata.json ownerId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_rep_person/metadata/db1c3994-dd04-410a-aa95-6516b7615246.metadata.json
[IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_base_city (1/1)#2] 2023-01-11 18:46:03,672 INFO org.apache.iceberg.util.LockManagers$InMemoryLockManager [] - acquired lock. entityId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_base_city/metadata/v749.metadata.json ownerId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_base_city/metadata/27a6b56d-7747-4a7b-9da1-694c5c6f8b3b.metadata.json
[IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_base_city (1/1)#2] 2023-01-11 18:46:03,672 INFO org.apache.iceberg.util.LockManagers$InMemoryLockManager [] - add lock heartbeat. entityId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_base_city/metadata/v749.metadata.json ownerId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_base_city/metadata/27a6b56d-7747-4a7b-9da1-694c5c6f8b3b.metadata.json
[IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_rep_person (1/1)#2] 2023-01-11 18:46:03,672 INFO org.apache.iceberg.util.LockManagers$InMemoryLockManager [] - add lock heartbeat. entityId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_rep_person/metadata/v756.metadata.json ownerId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_rep_person/metadata/db1c3994-dd04-410a-aa95-6516b7615246.metadata.json
[IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_rep_docsource_type (1/1)#2] 2023-01-11 18:46:03,674 INFO org.apache.iceberg.util.LockManagers$InMemoryLockManager [] - releasing lock. entityId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_rep_docsource_type/metadata/v749.metadata.json ownerId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_rep_docsource_type/metadata/22f4f9c5-35d2-467c-8263-d26fd6548017.metadata.json
[IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_rep_docsource_type (1/1)#2] 2023-01-11 18:46:03,674 INFO org.apache.iceberg.util.LockManagers$InMemoryLockManager [] - released lock. entityId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_rep_docsource_type/metadata/v749.metadata.json ownerId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_rep_docsource_type/metadata/22f4f9c5-35d2-467c-8263-d26fd6548017.metadata.json
[IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_rep_docsource_type (1/1)#2] 2023-01-11 18:46:03,674 INFO org.apache.iceberg.hadoop.HadoopTableOperations [] - Committed a new metadata file hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_rep_docsource_type/metadata/v749.metadata.json
[IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_base_province (1/1)#2] 2023-01-11 18:46:03,677 INFO org.apache.iceberg.util.LockManagers$InMemoryLockManager [] - releasing lock. entityId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_base_province/metadata/v749.metadata.json ownerId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_base_province/metadata/ced05720-8f32-49c9-884d-49b604290622.metadata.json
[IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_base_province (1/1)#2] 2023-01-11 18:46:03,677 INFO org.apache.iceberg.util.LockManagers$InMemoryLockManager [] - released lock. entityId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_base_province/metadata/v749.metadata.json ownerId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_base_province/metadata/ced05720-8f32-49c9-884d-49b604290622.metadata.json
[IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_base_province (1/1)#2] 2023-01-11 18:46:03,677 INFO org.apache.iceberg.hadoop.HadoopTableOperations [] - Committed a new metadata file hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_base_province/metadata/v749.metadata.json
[IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_base_city (1/1)#2] 2023-01-11 18:46:03,679 INFO org.apache.iceberg.util.LockManagers$InMemoryLockManager [] - releasing lock. entityId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_base_city/metadata/v749.metadata.json ownerId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_base_city/metadata/27a6b56d-7747-4a7b-9da1-694c5c6f8b3b.metadata.json
[IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_rep_person (1/1)#2] 2023-01-11 18:46:03,679 INFO org.apache.iceberg.util.LockManagers$InMemoryLockManager [] - releasing lock. entityId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_rep_person/metadata/v756.metadata.json ownerId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_rep_person/metadata/db1c3994-dd04-410a-aa95-6516b7615246.metadata.json
[IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_rep_person (1/1)#2] 2023-01-11 18:46:03,679 INFO org.apache.iceberg.util.LockManagers$InMemoryLockManager [] - released lock. entityId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_rep_person/metadata/v756.metadata.json ownerId:hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_rep_person/metadata/db1c3994-dd04-410a-aa95-6516b7615246.metadata.json
[IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_rep_person (1/1)#2] 2023-01-11 18:46:03,679 INFO org.apache.iceberg.hadoop.HadoopTableOperations [] - Committed a new metadata file hdfs://hadoop-node-0001:8022/uas/ods_dms_db/ods_t_rep_person/metadata/v756.metadata.json
[IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_base_city (1/1)#2] 2023-01-11 18:46:03,684 WARN org.apache.flink.runtime.taskmanager.Task [] - IcebergFilesCommitter -> Sink: IcebergSink hadoop.ods_dms_db.ods_t_base_city (1/1)#2 (5dec3e28251735766bb3eb423ca5a45c) switched from RUNNING to FAILED with failure cause: java.lang.NullPointerException
at org.apache.iceberg.util.LockManagers$InMemoryLockManager.release(LockManagers.java:235)
at org.apache.iceberg.hadoop.HadoopTableOperations.renameToFinal(HadoopTableOperations.java:377)
at org.apache.iceberg.hadoop.HadoopTableOperations.commit(HadoopTableOperations.java:159)
at org.apache.iceberg.SnapshotProducer.lambda$commit$2(SnapshotProducer.java:317)
at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:404)
at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:214)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:198)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:190)
at org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:295)
at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitOperation(IcebergFilesCommitter.java:312)
at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitDeltaTxn(IcebergFilesCommitter.java:276)
at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitUpToCheckpoint(IcebergFilesCommitter.java:218)
at org.apache.iceberg.flink.sink.IcebergFilesCommitter.notifyCheckpointComplete(IcebergFilesCommitter.java:188)
at org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.notifyCheckpointComplete(StreamOperatorWrapper.java:99)
at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.notifyCheckpointComplete(SubtaskCheckpointCoordinatorImpl.java:334)
at org.apache.flink.streaming.runtime.tasks.StreamTask.notifyCheckpointComplete(StreamTask.java:1171)
at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointCompleteAsync$10(StreamTask.java:1136)
at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointOperation$12(StreamTask.java:1159)
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:344)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:330)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:202)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:684)
at org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:639)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:623)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
at java.lang.Thread.run(Thread.java:748)
`
The code is as follows:
`
@VisibleForTesting
void acquireOnce(String entityId, String ownerId) {
InMemoryLockContent content = LOCKS.get(entityId);
if (content != null && content.expireMs() > System.currentTimeMillis()) {
throw new IllegalStateException(String.format("Lock for %s currently held by %s, expiration: %s",
entityId, content.ownerId(), content.expireMs()));
}
LOG.info("acquiring lock. entityId:{} ownerId:{}", entityId, ownerId);
long expiration = System.currentTimeMillis() + heartbeatTimeoutMs();
boolean succeed;
if (content == null) {
InMemoryLockContent previous = LOCKS.putIfAbsent(
entityId, new InMemoryLockContent(ownerId, expiration));
succeed = previous == null;
} else {
succeed = LOCKS.replace(entityId, content, new InMemoryLockContent(ownerId, expiration));
}
if (succeed) {
LOG.info("acquired lock. entityId:{} ownerId:{}", entityId, ownerId);
// cleanup old heartbeat
if (HEARTBEATS.containsKey(entityId)) {
HEARTBEATS.remove(entityId).cancel(false);
LOG.info("remove lock heartbeat. entityId:{} ownerId:{}", entityId, ownerId);
}
HEARTBEATS.put(entityId, scheduler().scheduleAtFixedRate(() -> {
InMemoryLockContent lastContent = LOCKS.get(entityId);
try {
long newExpiration = System.currentTimeMillis() + heartbeatTimeoutMs();
LOCKS.replace(entityId, lastContent, new InMemoryLockContent(ownerId, newExpiration));
} catch (NullPointerException e) {
throw new RuntimeException("Cannot heartbeat to a deleted lock " + entityId, e);
}
}, 0, heartbeatIntervalMs(), TimeUnit.MILLISECONDS));
LOG.info("add lock heartbeat. entityId:{} ownerId:{}", entityId, ownerId);
} else {
throw new IllegalStateException("Unable to acquire lock " + entityId);
}
}
@Override
public boolean release(String entityId, String ownerId) {
LOG.info("releasing lock. entityId:{} ownerId:{}", entityId, ownerId);
InMemoryLockContent currentContent = LOCKS.get(entityId);
if (currentContent == null) {
LOG.error("Cannot find lock for entity {}", entityId);
return false;
}
if (!currentContent.ownerId().equals(ownerId)) {
LOG.error("Cannot unlock {} by {}, current owner: {}", entityId, ownerId, currentContent.ownerId());
return false;
}
HEARTBEATS.remove(entityId).cancel(false);
LOCKS.remove(entityId);
LOG.info("released lock. entityId:{} ownerId:{}", entityId, ownerId);
return true;
}
`
HEARTBEATS.remove(entityId) is empty when the lock is released, strangely, 2023-01-11 18:46:03, 672 added entityId heartbeart, 2023-01-11 18:46:03, 684 can not find the corresponding heartbeat, the whole process is single-threaded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org