[jira] [Created] (STORM-1378) Windows File Handling Issues with Blobstore

["Chuck Burgess (JIRA)" <ji...@apache.org>]

Chuck Burgess created STORM-1378:
------------------------------------

             Summary: Windows File Handling Issues with Blobstore
                 Key: STORM-1378
                 URL: https://issues.apache.org/jira/browse/STORM-1378
             Project: Apache Storm
          Issue Type: Bug
         Environment: Win7 JDK1.7.0_45
            Reporter: Chuck Burgess
            Priority: Minor


Since the Blobstore merge to master (7029ae), several tests show consistent issues with file system interaction on Windows.

I first noticed it as several of these in backtype.storm.multilang-test.test-python:

{code}
30970 [Thread-8] ERROR b.s.event - Error when processing event

java.io.IOException: Unable to delete file: C:\Users\cburgess\AppData\Local\Temp\6d2e5448-3710-44c0-8e0a-8bfef14d065d\supervisor\tmp\fcc042a9-ddf4-498d-9114-a29626429310\stormconf.ser
	at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2279) ~[commons-io-2.4.jar:2.4]
	at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653) ~[commons-io-2.4.jar:2.4]
	at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535) ~[commons-io-2.4.jar:2.4]
	at org.apache.commons.io.FileUtils.moveDirectory(FileUtils.java:2756) ~[commons-io-2.4.jar:2.4]
	at backtype.storm.daemon.supervisor$fn__2392.invoke(supervisor.clj:1134) ~[classes/:?]
	at clojure.lang.MultiFn.invoke(MultiFn.java:243) ~[clojure-1.7.0.jar:?]
	at backtype.storm.daemon.supervisor$mk_synchronize_supervisor$this__2118.invoke(supervisor.clj:565) ~[classes/:?]
	at backtype.storm.event$event_manager$fn__5761.invoke(event.clj:40) [classes/:?]
	at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
	at java.lang.Thread.run(Thread.java:744) [?:1.7.0_45]
{code}


When running a full test build, the LocalizerTest and BlobStoreTest cases stop the build due to Windows file interaction errors:

{code}
Failed tests:
  LocalizerTest.testDirPaths:147 get local user dir doesn't return right value expected:<...c5-8a25-f0a888dc95e0[/usercache/]user1> but was:<...c5-8a25-f0a888dc95e0[\usercache\]user1>
Tests in error:
backtype.storm.blobstore.BlobStoreTest.testBasicLocalFs(backtype.storm.blobstore.BlobStoreTest)
  Run 1: BlobStoreTest.testBasicLocalFs:164->testBasic:313 ╗ Runtime java.nio.file.Dire...
  Run 2: BlobStoreTest.cleanup:74 ╗ IO Unable to delete file: \tmp\blob-store-test-0a66...

  BlobStoreTest.cleanup:74 ╗ IO Unable to delete file: \tmp\blob-store-test-9ada...
backtype.storm.blobstore.BlobStoreTest.testLocalFsWithAuth(backtype.storm.blobstore.BlobStoreTest)
  Run 1: BlobStoreTest.testLocalFsWithAuth:159->testWithAuthentication:227 ╗ Runtime ja...
  Run 2: BlobStoreTest.cleanup:74 ╗ IO Unable to delete file: \tmp\blob-store-test-594f...

backtype.storm.blobstore.BlobStoreTest.testMultipleLocalFs(backtype.storm.blobstore.BlobStoreTest)
  Run 1: BlobStoreTest.testMultipleLocalFs:169->testMultiple:398 ╗ AccessDenied C:\tmp\...
  Run 2: BlobStoreTest.cleanup:74 ╗ IO Unable to delete file: \tmp\blob-store-test-5356...

  LocalizerTest.testArchivesJar:260->testArchives:286 ╗ IO Error getting blobs
  LocalizerTest.testArchivesTar:254->testArchives:286 ╗ IO Error getting blobs
backtype.storm.localizer.LocalizerTest.testArchivesTarGz(backtype.storm.localizer.LocalizerTest)
  Run 1: LocalizerTest.testArchivesTarGz:248->testArchives:286 ╗ IO Error getting blobs
  Run 2: LocalizerTest.tearDown:125 ╗ IO Unable to delete file: \tmp\blob-store-localiz...

backtype.storm.localizer.LocalizerTest.testArchivesTgz(backtype.storm.localizer.LocalizerTest)
  Run 1: LocalizerTest.testArchivesTgz:236->testArchives:286 ╗ IO Error getting blobs
  Run 2: LocalizerTest.tearDown:125 ╗ IO Unable to delete file: \tmp\blob-store-localiz...

  LocalizerTest.testArchivesZip:242->testArchives:286 ╗ IO Error getting blobs
  LocalizerTest.testBasic:363 ╗ IO Error getting blobs
  LocalizerTest.testMultipleKeysOneUser:435 ╗ IO Error getting blobs
  LocalizerTest.testMultipleUsers:563 ╗ IO Error getting blobs
  LocalizerTest.testUpdate:637 ╗ IO Error getting blobs

Tests run: 95, Failures: 1, Errors: 13, Skipped: 0
{code}

Most of these are also due to "Unable to delete file".  They also all seem to be regarding stormconf.ser.

Is it possible that some process/thread that created these stormconf.ser files has not released/closed their own file interactions before the storm code tries to delete them?

Because 7029ae looks to be the entire feature squashed into one commit, I cannot bisect any deeper.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)