You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user-zh@flink.apache.org by guo liu <lg...@gmail.com> on 2019/09/19 10:02:23 UTC
Flink BlobServerConnection NoSuchFileException
hi,
我们在使用flink-stream时遇到了如下问题，有谁可以帮助一下吗？怎么避免该问题呢？
flink版本：flink-1.8.1 standalone cluster
node4是我们集群中的一个slave节点，我们的任务运行了差不多1个月才出现的这个问题。这个异常导致任务全部处于失败然后恢复失败的循环中，目前不知道什么原因造成的，查看了
/tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/文件夹是空的。
谢谢。
2019-09-19 16:54:34,328 ERROR
org.apache.flink.runtime.blob.BlobServerConnection - GET
operation failed for BLOB
ba2fb0924f20a093cff50b27aa8ea6c1/p-105986c82d33c1ed56863caabc5f9b6517bc16c3-2122fe23e729d1ea1d0a4f13f16d1e28
from /node04.
java.nio.file.NoSuchFileException:
/tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/temp-03506073
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409)
at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
at java.nio.file.Files.move(Files.java:1395)
at
org.apache.flink.runtime.blob.BlobUtils.moveTempFileToStore(BlobUtils.java:429)
at
org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:497)
at
org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231)
at
org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117)
2019-09-19 16:54:34,330 ERROR
org.apache.flink.runtime.blob.BlobServerConnection - GET
operation failed for BLOB
ba2fb0924f20a093cff50b27aa8ea6c1/p-105986c82d33c1ed56863caabc5f9b6517bc16c3-2122fe23e729d1ea1d0a4f13f16d1e28
from /node04 .
java.nio.file.NoSuchFileException:
/tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/temp-03506074
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409)
at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
at java.nio.file.Files.move(Files.java:1395)
at
org.apache.flink.runtime.blob.BlobUtils.moveTempFileToStore(BlobUtils.java:429)
at
org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:497)
at
org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231)
at
org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117)
2019-09-19 16:54:34,331 ERROR
org.apache.flink.runtime.blob.BlobServerConnection - GET
operation failed for BLOB
ba2fb0924f20a093cff50b27aa8ea6c1/p-105986c82d33c1ed56863caabc5f9b6517bc16c3-2122fe23e729d1ea1d0a4f13f16d1e28
from /node04 .
java.nio.file.NoSuchFileException:
/tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/temp-03506075
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409)
at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
at java.nio.file.Files.move(Files.java:1395)
at
org.apache.flink.runtime.blob.BlobUtils.moveTempFileToStore(BlobUtils.java:429)
at
org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:497)
at
org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231)
at
org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117)
2019-09-19 16:54:34,332 ERROR
org.apache.flink.runtime.blob.BlobServerConnection - GET
operation failed for BLOB
ba2fb0924f20a093cff50b27aa8ea6c1/p-105986c82d33c1ed56863caabc5f9b6517bc16c3-2122fe23e729d1ea1d0a4f13f16d1e28
from /node04 .
java.nio.file.NoSuchFileException:
/tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/temp-03506076
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409)
at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
at java.nio.file.Files.move(Files.java:1395)
at
org.apache.flink.runtime.blob.BlobUtils.moveTempFileToStore(BlobUtils.java:429)
at
org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:497)
at
org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231)
at
org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117)
Re: Flink BlobServerConnection NoSuchFileException
Posted by Dian Fu <di...@gmail.com>.
我猜测原因是blob server目录里存的blob文件被删了，所以/tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/temp-03506073目录为空，具体可以参考如下代码：
https://github.com/apache/flink/blob/1a81982ba864d72fda3d161fc82ca1572333ba11/flink-runtime/src/main/java/org/apache/flink/runtime/blob/FileSystemBlobStore.java#L129 <https://github.com/apache/flink/blob/1a81982ba864d72fda3d161fc82ca1572333ba11/flink-runtime/src/main/java/org/apache/flink/runtime/blob/FileSystemBlobStore.java#L129>
有一个类似问题，你可以看一下是否有帮助：
http://mail-archives.apache.org/mod_mbox/flink-user/201810.mbox/%3c784feda8-9a82-c938-8dbe-d7e8e2650916@data-artisans.com%3e <http://mail-archives.apache.org/mod_mbox/flink-user/201810.mbox/%3C784feda8-9a82-c938-8dbe-d7e8e2650916@data-artisans.com%3E>
Regards,
Dian
> 在 2019年9月19日，下午6:02，guo liu <lg...@gmail.com> 写道：
>
> hi,
> 我们在使用flink-stream时遇到了如下问题，有谁可以帮助一下吗？怎么避免该问题呢？
>
> flink版本：flink-1.8.1 standalone cluster
>
> node4是我们集群中的一个slave节点，我们的任务运行了差不多1个月才出现的这个问题。这个异常导致任务全部处于失败然后恢复失败的循环中，目前不知道什么原因造成的，查看了
> /tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/文件夹是空的。
>
> 谢谢。
>
> 2019-09-19 16:54:34,328 ERROR
> org.apache.flink.runtime.blob.BlobServerConnection - GET
> operation failed for BLOB
> ba2fb0924f20a093cff50b27aa8ea6c1/p-105986c82d33c1ed56863caabc5f9b6517bc16c3-2122fe23e729d1ea1d0a4f13f16d1e28
> from /node04.
> java.nio.file.NoSuchFileException:
> /tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/temp-03506073
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409)
> at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
> at java.nio.file.Files.move(Files.java:1395)
> at
> org.apache.flink.runtime.blob.BlobUtils.moveTempFileToStore(BlobUtils.java:429)
> at
> org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:497)
> at
> org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231)
> at
> org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117)
> 2019-09-19 16:54:34,330 ERROR
> org.apache.flink.runtime.blob.BlobServerConnection - GET
> operation failed for BLOB
> ba2fb0924f20a093cff50b27aa8ea6c1/p-105986c82d33c1ed56863caabc5f9b6517bc16c3-2122fe23e729d1ea1d0a4f13f16d1e28
> from /node04 .
> java.nio.file.NoSuchFileException:
> /tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/temp-03506074
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409)
> at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
> at java.nio.file.Files.move(Files.java:1395)
> at
> org.apache.flink.runtime.blob.BlobUtils.moveTempFileToStore(BlobUtils.java:429)
> at
> org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:497)
> at
> org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231)
> at
> org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117)
> 2019-09-19 16:54:34,331 ERROR
> org.apache.flink.runtime.blob.BlobServerConnection - GET
> operation failed for BLOB
> ba2fb0924f20a093cff50b27aa8ea6c1/p-105986c82d33c1ed56863caabc5f9b6517bc16c3-2122fe23e729d1ea1d0a4f13f16d1e28
> from /node04 .
> java.nio.file.NoSuchFileException:
> /tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/temp-03506075
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409)
> at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
> at java.nio.file.Files.move(Files.java:1395)
> at
> org.apache.flink.runtime.blob.BlobUtils.moveTempFileToStore(BlobUtils.java:429)
> at
> org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:497)
> at
> org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231)
> at
> org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117)
> 2019-09-19 16:54:34,332 ERROR
> org.apache.flink.runtime.blob.BlobServerConnection - GET
> operation failed for BLOB
> ba2fb0924f20a093cff50b27aa8ea6c1/p-105986c82d33c1ed56863caabc5f9b6517bc16c3-2122fe23e729d1ea1d0a4f13f16d1e28
> from /node04 .
> java.nio.file.NoSuchFileException:
> /tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/temp-03506076
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409)
> at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
> at java.nio.file.Files.move(Files.java:1395)
> at
> org.apache.flink.runtime.blob.BlobUtils.moveTempFileToStore(BlobUtils.java:429)
> at
> org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:497)
> at
> org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231)
> at
> org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117)