You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@storm.apache.org by "Matt Wissman (JIRA)" <ji...@apache.org> on 2016/12/29 15:42:58 UTC
[jira] [Commented] (STORM-2095) Nimbus dies and never recovers due
to java.nio.file.DirectoryNotEmptyException
[ https://issues.apache.org/jira/browse/STORM-2095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15785548#comment-15785548 ]
Matt Wissman commented on STORM-2095:
-------------------------------------
When nimbus dies while a blob is being created, the temp file is left in the directory. When Nimbus starts up and tries to delete the blob, it's looking for a file named after the blob name. That file doesn't exist, and when the delete of the blob folder is attempted, it fails.
To fix this, when deleting the folder for the blob, this will also delete any partially uploaded temp files.
I created a pull request for this: https://github.com/apache/storm/pull/1848
> Nimbus dies and never recovers due to java.nio.file.DirectoryNotEmptyException
> ------------------------------------------------------------------------------
>
> Key: STORM-2095
> URL: https://issues.apache.org/jira/browse/STORM-2095
> Project: Apache Storm
> Issue Type: Bug
> Components: blobstore
> Affects Versions: 1.0.0, 1.0.1, 1.0.2, 1.1.0, 1.0.3
> Reporter: Abid Mohammed
> Priority: Critical
> Time Spent: 10m
> Remaining Estimate: 0h
>
> To Recreate:
> --------------------------------------
> 1) Create a blobstore key for a large file (1 or 2 GB). Size of the file does not matter if nimbus can be killed while the blob is being created.
> 2) while the blob is being created, restart nimbus (this is easiest way to regenerate, there can be various reasons due to which a blob couldn't be successfully created in nimbus)
> 3) When nimbus tries to start on restart, it will keep dying due to DirectoryNotEmptyException and never come up.
> Expected Behavior
> --------------------------------------
> Partial blobstore key is deleted cleanly and doesn't affect nimbus.
> The actual, incorrect behavior.
> --------------------------------------
> 2016-09-14 15:07:48.518 o.a.s.zookeeper [INFO] Queued up for leader lock.
> 2016-09-14 15:07:48.576 o.a.s.zookeeper [INFO] xxx gained leadership
> 2016-09-14 15:07:48.581 o.a.s.d.nimbus [ERROR] Error on initialization of server service-handler
> java.lang.RuntimeException: java.nio.file.DirectoryNotEmptyException: /opt/storm/storm-local/blobs/955/some_big_file
> at org.apache.storm.blobstore.LocalFsBlobStore.deleteBlob(LocalFsBlobStore.java:229)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
> at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
> at org.apache.storm.daemon.nimbus$setup_blobstore.invoke(nimbus.clj:1196)
> at org.apache.storm.daemon.nimbus$fn__7064$exec_fn__2461__auto____7065.invoke(nimbus.clj:1416)
> at clojure.lang.AFn.applyToHelper(AFn.java:156)
> at clojure.lang.AFn.applyTo(AFn.java:144)
> at clojure.core$apply.invoke(core.clj:630)
> at org.apache.storm.daemon.nimbus$fn__7064$service_handler__7308.doInvoke(nimbus.clj:1358)
> at clojure.lang.RestFn.invoke(RestFn.java:421)
> at org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:2206)
> at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2239)
> at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2262)
> at clojure.lang.AFn.applyToHelper(AFn.java:152)
> at clojure.lang.AFn.applyTo(AFn.java:144)
> at org.apache.storm.daemon.nimbus.main(Unknown Source)
> Caused by: java.nio.file.DirectoryNotEmptyException: /opt/storm/storm-local/blobs/955/some_big_file
> at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
> at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
> at java.nio.file.Files.deleteIfExists(Files.java:1165)
> at org.apache.storm.blobstore.FileBlobStoreImpl.delete(FileBlobStoreImpl.java:239)
> at org.apache.storm.blobstore.FileBlobStoreImpl.deleteKey(FileBlobStoreImpl.java:178)
> at org.apache.storm.blobstore.LocalFsBlobStore.deleteBlob(LocalFsBlobStore.java:226)
> ... 19 more
> 2016-09-14 15:07:48.588 o.a.s.util [ERROR] Halting process: ("Error on initialization")
> java.lang.RuntimeException: ("Error on initialization")
> at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
> at clojure.lang.RestFn.invoke(RestFn.java:423)
> at org.apache.storm.daemon.nimbus$fn__7064$service_handler__7308.doInvoke(nimbus.clj:1358)
> at clojure.lang.RestFn.invoke(RestFn.java:421)
> at org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:2206)
> at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2239)
> at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2262)
> at clojure.lang.AFn.applyToHelper(AFn.java:152)
> at clojure.lang.AFn.applyTo(AFn.java:144)
> at org.apache.storm.daemon.nimbus.main(Unknown Source)
> [root]# ls -l /opt/storm/storm-local/blobs/955/some_big_file
> total 591060
> -rw-r--r-- 1 storm storm 605241344 Sep 14 15:07 1473865562841.tmp
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)