You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Stephan Ewen (JIRA)" <ji...@apache.org> on 2016/12/15 15:56:58 UTC
[jira] [Commented] (FLINK-5345) IOManager failed to properly clean
up temp file directory
[ https://issues.apache.org/jira/browse/FLINK-5345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15751722#comment-15751722 ]
Stephan Ewen commented on FLINK-5345:
-------------------------------------
I think that is a problem of {{org.apache.commons.io.FileUtils}}: When someone concurrently works on the directory, the delete fails.
We should have our own utility method for recursive directory that retries listing and deleting contained files to be safe against concurrent deletes by other services.
> IOManager failed to properly clean up temp file directory
> ---------------------------------------------------------
>
> Key: FLINK-5345
> URL: https://issues.apache.org/jira/browse/FLINK-5345
> Project: Flink
> Issue Type: Bug
> Affects Versions: 1.1.3
> Reporter: Robert Metzger
> Labels: simplex, starter
>
> While testing 1.1.3 RC3, I have the following message in my log:
> {code}
> 2016-12-15 14:46:05,450 INFO org.apache.flink.streaming.runtime.tasks.StreamTask - Timer service is shutting down.
> 2016-12-15 14:46:05,452 INFO org.apache.flink.runtime.taskmanager.Task - Source: control events generator (29/40) (73915a232ba09e642f9dff92f8c8773a) switched from CANCELING to CANCELED.
> 2016-12-15 14:46:05,452 INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for Source: control events generator (29/40) (73915a232ba09e642f9dff92f8c8773a).
> 2016-12-15 14:46:05,454 INFO org.apache.flink.yarn.YarnTaskManager - Un-registering task and sending final execution state CANCELED to JobManager for task Source: control events genera
> tor (73915a232ba09e642f9dff92f8c8773a)
> 2016-12-15 14:46:40,609 INFO org.apache.flink.yarn.YarnTaskManagerRunner - RECEIVED SIGNAL 15: SIGTERM. Shutting down as requested.
> 2016-12-15 14:46:40,611 INFO org.apache.flink.runtime.blob.BlobCache - Shutting down BlobCache
> 2016-12-15 14:46:40,724 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.240.0.34:33635] has failed, address is now gated for [5000] ms.
> Reason is: [Disassociated].
> 2016-12-15 14:46:40,808 ERROR org.apache.flink.runtime.io.disk.iomanager.IOManager - IOManager failed to properly clean up temp file directory: /yarn/nm/usercache/robert/appcache/application_148129128
> 9979_0024/flink-io-f0ff3f66-b9e2-4560-881f-2ab43bc448b5
> java.lang.IllegalArgumentException: /yarn/nm/usercache/robert/appcache/application_1481291289979_0024/flink-io-f0ff3f66-b9e2-4560-881f-2ab43bc448b5/62e14e1891fe1e334c921dfd19a32a84/StreamMap_11_24/dummy_state does not exist
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1637)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.flink.runtime.io.disk.iomanager.IOManager.shutdown(IOManager.java:109)
> at org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync.shutdown(IOManagerAsync.java:185)
> at org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync$1.run(IOManagerAsync.java:105)
> {code}
> This was the last message logged from that machine. I suspect two threads are trying to clean up the directories during shutdown?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)