[jira] [Created] (STORM-2095) Nimbus dies and never recovers due to java.nio.file.DirectoryNotEmptyException

["Abid Mohammed (JIRA)" <ji...@apache.org>]

Abid Mohammed created STORM-2095:
------------------------------------

             Summary: Nimbus dies and never recovers due to java.nio.file.DirectoryNotEmptyException
                 Key: STORM-2095
                 URL: https://issues.apache.org/jira/browse/STORM-2095
             Project: Apache Storm
          Issue Type: Bug
          Components: blobstore
    Affects Versions: 1.0.0, 1.0.1, 1.0.2, 1.1.0, 1.0.3
            Reporter: Abid Mohammed
            Priority: Critical


To Recreate:
--------------------------------------
1) Create a blobstore key for a large file (1 or 2 GB). Size of the file does not matter if nimbus can be killed while the blob is being created.
2) while the blob is being created, restart nimbus (this is easiest way to regenerate, there can be various reasons due to which a blob couldn't be successfully created in nimbus)
3) When nimbus tries to start on restart, it will keep dying due to DirectoryNotEmptyException and never come up.

Expected Behavior
--------------------------------------
Partial blobstore key is deleted cleanly and doesn't affect nimbus.

The actual, incorrect behavior.
--------------------------------------
2016-09-14 15:07:48.518 o.a.s.zookeeper [INFO] Queued up for leader lock.
2016-09-14 15:07:48.576 o.a.s.zookeeper [INFO] xxx gained leadership
2016-09-14 15:07:48.581 o.a.s.d.nimbus [ERROR] Error on initialization of server service-handler
java.lang.RuntimeException: java.nio.file.DirectoryNotEmptyException: /opt/storm/storm-local/blobs/955/some_big_file
	at org.apache.storm.blobstore.LocalFsBlobStore.deleteBlob(LocalFsBlobStore.java:229)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
	at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
	at org.apache.storm.daemon.nimbus$setup_blobstore.invoke(nimbus.clj:1196)
	at org.apache.storm.daemon.nimbus$fn__7064$exec_fn__2461__auto____7065.invoke(nimbus.clj:1416)
	at clojure.lang.AFn.applyToHelper(AFn.java:156)
	at clojure.lang.AFn.applyTo(AFn.java:144)
	at clojure.core$apply.invoke(core.clj:630)
	at org.apache.storm.daemon.nimbus$fn__7064$service_handler__7308.doInvoke(nimbus.clj:1358)
	at clojure.lang.RestFn.invoke(RestFn.java:421)
	at org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:2206)
	at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2239)
	at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2262)
	at clojure.lang.AFn.applyToHelper(AFn.java:152)
	at clojure.lang.AFn.applyTo(AFn.java:144)
	at org.apache.storm.daemon.nimbus.main(Unknown Source)
Caused by: java.nio.file.DirectoryNotEmptyException: /opt/storm/storm-local/blobs/955/some_big_file
	at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
	at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
	at java.nio.file.Files.deleteIfExists(Files.java:1165)
	at org.apache.storm.blobstore.FileBlobStoreImpl.delete(FileBlobStoreImpl.java:239)
	at org.apache.storm.blobstore.FileBlobStoreImpl.deleteKey(FileBlobStoreImpl.java:178)
	at org.apache.storm.blobstore.LocalFsBlobStore.deleteBlob(LocalFsBlobStore.java:226)
	... 19 more
2016-09-14 15:07:48.588 o.a.s.util [ERROR] Halting process: ("Error on initialization")
java.lang.RuntimeException: ("Error on initialization")
	at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
	at clojure.lang.RestFn.invoke(RestFn.java:423)
	at org.apache.storm.daemon.nimbus$fn__7064$service_handler__7308.doInvoke(nimbus.clj:1358)
	at clojure.lang.RestFn.invoke(RestFn.java:421)
	at org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:2206)
	at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2239)
	at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2262)
	at clojure.lang.AFn.applyToHelper(AFn.java:152)
	at clojure.lang.AFn.applyTo(AFn.java:144)
	at org.apache.storm.daemon.nimbus.main(Unknown Source)


[root]# ls -l  /opt/storm/storm-local/blobs/955/some_big_file
total 591060
-rw-r--r-- 1 storm storm 605241344 Sep 14 15:07 1473865562841.tmp



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)