You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@storm.apache.org by "Jungtaek Lim (JIRA)" <ji...@apache.org> on 2017/01/03 14:18:58 UTC
[jira] [Resolved] (STORM-2095) Nimbus dies and never recovers due
to java.nio.file.DirectoryNotEmptyException
[ https://issues.apache.org/jira/browse/STORM-2095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jungtaek Lim resolved STORM-2095.
---------------------------------
Resolution: Fixed
Assignee: Matt Wissman
Fix Version/s: 1.0.3
1.1.0
2.0.0
Thanks [~mwissman], I merged into master, 1.x, 1.0.x branches.
> Nimbus dies and never recovers due to java.nio.file.DirectoryNotEmptyException
> ------------------------------------------------------------------------------
>
> Key: STORM-2095
> URL: https://issues.apache.org/jira/browse/STORM-2095
> Project: Apache Storm
> Issue Type: Bug
> Components: blobstore
> Affects Versions: 1.0.0, 1.0.1, 1.0.2, 1.1.0, 1.0.3
> Reporter: Abid Mohammed
> Assignee: Matt Wissman
> Priority: Critical
> Fix For: 2.0.0, 1.1.0, 1.0.3
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> To Recreate:
> --------------------------------------
> 1) Create a blobstore key for a large file (1 or 2 GB). Size of the file does not matter if nimbus can be killed while the blob is being created.
> 2) while the blob is being created, restart nimbus (this is easiest way to regenerate, there can be various reasons due to which a blob couldn't be successfully created in nimbus)
> 3) When nimbus tries to start on restart, it will keep dying due to DirectoryNotEmptyException and never come up.
> Expected Behavior
> --------------------------------------
> Partial blobstore key is deleted cleanly and doesn't affect nimbus.
> The actual, incorrect behavior.
> --------------------------------------
> 2016-09-14 15:07:48.518 o.a.s.zookeeper [INFO] Queued up for leader lock.
> 2016-09-14 15:07:48.576 o.a.s.zookeeper [INFO] xxx gained leadership
> 2016-09-14 15:07:48.581 o.a.s.d.nimbus [ERROR] Error on initialization of server service-handler
> java.lang.RuntimeException: java.nio.file.DirectoryNotEmptyException: /opt/storm/storm-local/blobs/955/some_big_file
> at org.apache.storm.blobstore.LocalFsBlobStore.deleteBlob(LocalFsBlobStore.java:229)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
> at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
> at org.apache.storm.daemon.nimbus$setup_blobstore.invoke(nimbus.clj:1196)
> at org.apache.storm.daemon.nimbus$fn__7064$exec_fn__2461__auto____7065.invoke(nimbus.clj:1416)
> at clojure.lang.AFn.applyToHelper(AFn.java:156)
> at clojure.lang.AFn.applyTo(AFn.java:144)
> at clojure.core$apply.invoke(core.clj:630)
> at org.apache.storm.daemon.nimbus$fn__7064$service_handler__7308.doInvoke(nimbus.clj:1358)
> at clojure.lang.RestFn.invoke(RestFn.java:421)
> at org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:2206)
> at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2239)
> at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2262)
> at clojure.lang.AFn.applyToHelper(AFn.java:152)
> at clojure.lang.AFn.applyTo(AFn.java:144)
> at org.apache.storm.daemon.nimbus.main(Unknown Source)
> Caused by: java.nio.file.DirectoryNotEmptyException: /opt/storm/storm-local/blobs/955/some_big_file
> at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
> at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
> at java.nio.file.Files.deleteIfExists(Files.java:1165)
> at org.apache.storm.blobstore.FileBlobStoreImpl.delete(FileBlobStoreImpl.java:239)
> at org.apache.storm.blobstore.FileBlobStoreImpl.deleteKey(FileBlobStoreImpl.java:178)
> at org.apache.storm.blobstore.LocalFsBlobStore.deleteBlob(LocalFsBlobStore.java:226)
> ... 19 more
> 2016-09-14 15:07:48.588 o.a.s.util [ERROR] Halting process: ("Error on initialization")
> java.lang.RuntimeException: ("Error on initialization")
> at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
> at clojure.lang.RestFn.invoke(RestFn.java:423)
> at org.apache.storm.daemon.nimbus$fn__7064$service_handler__7308.doInvoke(nimbus.clj:1358)
> at clojure.lang.RestFn.invoke(RestFn.java:421)
> at org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:2206)
> at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2239)
> at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2262)
> at clojure.lang.AFn.applyToHelper(AFn.java:152)
> at clojure.lang.AFn.applyTo(AFn.java:144)
> at org.apache.storm.daemon.nimbus.main(Unknown Source)
> [root]# ls -l /opt/storm/storm-local/blobs/955/some_big_file
> total 591060
> -rw-r--r-- 1 storm storm 605241344 Sep 14 15:07 1473865562841.tmp
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)