You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/09/12 21:35:36 UTC

[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_67) - Build # 11232 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11232/
Java: 64bit/jdk1.7.0_67 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

All tests passed

Build Log:
[...truncated 20308 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:114: The following files contain @author tags, tabs, svn keywords or nocommits:
* solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java

Total time: 101 minutes 32 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.7.0_67 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.9.0-ea-b28) - Build # 11247 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11247/
Java: 32bit/jdk1.9.0-ea-b28 -server -XX:+UseConcMarkSweepGC

5 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestZkChroot

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=5004, name=coreLoadExecutor-2689-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=5004, name=coreLoadExecutor-2689-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
	at java.lang.Thread.getStackTrace(Thread.java:1552)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@c2af4e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@13773d)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:699)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:696)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:712)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	... 1 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestZkChroot

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=5020, name=coreLoadExecutor-2696-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=5020, name=coreLoadExecutor-2696-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
	at java.lang.Thread.getStackTrace(Thread.java:1552)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@1ce4d3a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@13a71f6)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:699)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:696)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:712)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	... 1 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestZkChroot

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=5049, name=coreLoadExecutor-2706-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=5049, name=coreLoadExecutor-2706-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
	at java.lang.Thread.getStackTrace(Thread.java:1552)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@114eb46 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@fa37bf)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:699)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:696)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:712)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	... 1 more


FAILED:  org.apache.solr.core.CachingDirectoryFactoryTest.stressTest

Error Message:
Captured an uncaught exception in thread: Thread[id=1436, name=Thread-630, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1436, name=Thread-630, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path9" "write")
	at __randomizedtesting.SeedInfo.seed([D4A7220426F8F346]:0)
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
	at java.security.AccessController.checkPermission(AccessController.java:894)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:541)
	at java.lang.SecurityManager.checkWrite(SecurityManager.java:971)
	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
	at java.nio.file.Files.createDirectory(Files.java:673)
	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:780)
	at java.nio.file.Files.createDirectories(Files.java:726)
	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)


FAILED:  org.apache.solr.core.RAMDirectoryFactoryTest.test

Error Message:
access denied ("java.io.FilePermission" "/fake/path" "write")

Stack Trace:
java.security.AccessControlException: access denied ("java.io.FilePermission" "/fake/path" "write")
	at __randomizedtesting.SeedInfo.seed([D4A7220426F8F346:5CF31DDE88049EBE]:0)
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
	at java.security.AccessController.checkPermission(AccessController.java:894)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:541)
	at java.lang.SecurityManager.checkWrite(SecurityManager.java:971)
	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
	at java.nio.file.Files.createDirectory(Files.java:673)
	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:780)
	at java.nio.file.Files.createDirectories(Files.java:726)
	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
	at org.apache.solr.core.RAMDirectoryFactoryTest.dotestOpenReturnsTheSameForSamePath(RAMDirectoryFactoryTest.java:46)
	at org.apache.solr.core.RAMDirectoryFactoryTest.test(RAMDirectoryFactoryTest.java:33)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:484)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 10846 lines...]
   [junit4] Suite: org.apache.solr.core.RAMDirectoryFactoryTest
   [junit4]   2> 278227 T758 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@b481d2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1f4dafa assuming 'simple'
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=RAMDirectoryFactoryTest -Dtests.method=test -Dtests.seed=D4A7220426F8F346 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale= -Dtests.timezone=Indian/Comoro -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.01s J1 | RAMDirectoryFactoryTest.test <<<
   [junit4]    > Throwable #1: java.security.AccessControlException: access denied ("java.io.FilePermission" "/fake/path" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([D4A7220426F8F346:5CF31DDE88049EBE]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:894)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:541)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:971)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:673)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:780)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:726)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.RAMDirectoryFactoryTest.dotestOpenReturnsTheSameForSamePath(RAMDirectoryFactoryTest.java:46)
   [junit4]    > 	at org.apache.solr.core.RAMDirectoryFactoryTest.test(RAMDirectoryFactoryTest.java:33)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=DefaultSimilarity, locale=, timezone=Indian/Comoro
   [junit4]   2> NOTE: Linux 3.13.0-35-generic i386/Oracle Corporation 1.9.0-ea (32-bit)/cpus=8,threads=1,free=73561656,total=185192448
   [junit4]   2> NOTE: All tests run in this JVM: [CircularListTest, TestPhraseSuggestions, TestRandomMergePolicy, MigrateRouteKeyTest, NumericFieldsTest, QueryElevationComponentTest, TestIndexingPerformance, HttpPartitionTest, InfoHandlerTest, TestStressLucene, SuggesterFSTTest, TestCSVLoader, DateRangeFieldTest, TestFieldSortValues, BinaryUpdateRequestHandlerTest, SSLMigrationTest, BlockDirectoryTest, TestIndexSearcher, TestWriterPerf, TestCloudManagedSchema, TestCloudSchemaless, RAMDirectoryFactoryTest]
   [junit4] Completed on J1 in 0.04s, 1 test, 1 error <<< FAILURES!

[...truncated 140 lines...]
   [junit4] Suite: org.apache.solr.core.CachingDirectoryFactoryTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.core.CachingDirectoryFactoryTest-D4A7220426F8F346-001/init-core-data-001
   [junit4]   2> 534103 T1435 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 534105 T1435 oas.SolrTestCaseJ4.setUp ###Starting stressTest
   [junit4]   2> 534137 T1436 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@1a3c0a1 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@9eca9a assuming 'simple'
   [junit4]   2> 534139 T1445 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@5ebc9f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@163b6a4 assuming 'simple'
   [junit4]   2> 534139 T1440 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@1d37b40 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@ee717b assuming 'simple'
   [junit4]   2> 534140 T1443 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@18b2471 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@b62e4a assuming 'simple'
   [junit4]   2> 534140 T1441 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@12938aa lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@143291 assuming 'simple'
   [junit4]   2> 534140 T1439 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@17928c5 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4422fe assuming 'simple'
   [junit4]   2> 534140 T1438 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@11e722f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@237814 assuming 'simple'
   [junit4]   2> 534141 T1437 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@1c05e41 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@fd547a assuming 'simple'
   [junit4]   2> 37 T1441 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-635,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path9" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([D4A7220426F8F346]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:894)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:541)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:971)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:673)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:780)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:726)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 534141 T1446 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@765cc8 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@14b56f3 assuming 'simple'
   [junit4]   2> 534141 T1442 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@d329d0 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1ee23eb assuming 'simple'
   [junit4]   2> 534142 T1444 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@3c276a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1012d51 assuming 'simple'
   [junit4]   2> 37 T1436 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-630,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path9" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([D4A7220426F8F346]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:894)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:541)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:971)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:673)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:780)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:726)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 37 T1443 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-637,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path9" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([D4A7220426F8F346]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:894)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:541)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:971)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:673)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:780)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:726)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 37 T1439 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-633,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path9" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([D4A7220426F8F346]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:894)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:541)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:971)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:673)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:780)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:726)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 39 T1442 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-636,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path9" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([D4A7220426F8F346]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:894)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:541)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:971)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:673)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:780)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:726)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 39 T1444 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-638,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path9" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([D4A7220426F8F346]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:894)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:541)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:971)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:673)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:780)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:726)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 39 T1445 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-639,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path9" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([D4A7220426F8F346]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:894)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:541)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:971)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:673)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:780)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:726)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 38 T1446 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-640,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path9" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([D4A7220426F8F346]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:894)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:541)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:971)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:673)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:780)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:726)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 38 T1437 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-631,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path9" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([D4A7220426F8F346]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:894)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:541)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:971)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:673)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:780)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:726)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 38 T1438 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-632,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path9" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([D4A7220426F8F346]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:894)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:541)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:971)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:673)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:780)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:726)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 37 T1440 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-634,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path9" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([D4A7220426F8F346]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:894)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:541)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:971)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:673)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:780)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:726)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 542111 T1453 oasc.CachingDirectoryFactory.close Closing RAMDirectoryFactory - 0 directories currently being tracked
   [junit4]   2> 542268 T1435 oas.SolrTestCaseJ4.tearDown ###Ending stressTest
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CachingDirectoryFactoryTest -Dtests.method=stressTest -Dtests.seed=D4A7220426F8F346 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_SY -Dtests.timezone=Pacific/Galapagos -Dtests.file.encoding=UTF-8
   [junit4] ERROR   8.20s J0 | CachingDirectoryFactoryTest.stressTest <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1436, name=Thread-630, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path9" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([D4A7220426F8F346]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:894)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:541)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:971)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:673)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:780)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:726)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1441, name=Thread-635, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path9" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([D4A7220426F8F346]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:894)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:541)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:971)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:673)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:780)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:726)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1443, name=Thread-637, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path9" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([D4A7220426F8F346]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:894)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:541)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:971)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:673)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:780)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:726)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #4: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1440, name=Thread-634, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path9" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([D4A7220426F8F346]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:894)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:541)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:971)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:673)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:780)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:726)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #5: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1439, name=Thread-633, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path9" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([D4A7220426F8F346]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:894)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:541)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:971)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:673)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:780)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:726)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #6: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1438, name=Thread-632, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path9" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([D4A7220426F8F346]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:894)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:541)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:971)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:673)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:780)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:726)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #7: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1437, name=Thread-631, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path9" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([D4A7220426F8F346]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:894)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:541)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:971)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:673)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:780)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:726)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #8: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1446, name=Thread-640, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path9" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([D4A7220426F8F346]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:894)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:541)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:971)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:673)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:780)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:726)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #9: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1442, name=Thread-636, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path9" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([D4A7220426F8F346]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:894)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:541)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:971)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:673)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:780)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:726)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #10: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1444, name=Thread-638, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path9" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([D4A7220426F8F346]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:894)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:541)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:971)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:673)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:780)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:726)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #11: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1445, name=Thread-639, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path9" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([D4A7220426F8F346]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:894)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:541)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:971)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:673)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:780)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:726)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 542303 T1435 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{}, sim=DefaultSimilarity, locale=ar_SY, timezone=Pacific/Galapagos
   [junit4]   2> NOTE: Linux 3.13.0-35-generic i386/Oracle Corporation 1.9.0-ea (32-bit)/cpus=8,threads=1,free=77364808,total=192655360
   [junit4]   2> NOTE: All tests run in this JVM: [XsltUpdateRequestHandlerTest, HdfsBasicDistributedZkTest, TestSearchPerf, OverseerStatusTest, TestSolr4Spatial2, DirectSolrSpellCheckerTest, PeerSyncTest, TestSolrDeletionPolicy2, TestFastLRUCache, CursorMarkTest, TestSuggestSpellingConverter, TestSolr4Spatial, DistributedFacetPivotLargeTest, SpellCheckComponentTest, TestParamSet, DistributedSpellCheckComponentTest, TestFastWriter, TestMiniSolrCloudCluster, TestSchemaVersionResource, TestExceedMaxTermLength, TestPartialUpdateDeduplication, ModifyConfFileTest, RequiredFieldsTest, SuggesterWFSTTest, TestMaxScoreQueryParser, SpellPossibilityIteratorTest, TestSolrJ, EchoParamsTest, ClusterStateUpdateTest, ResponseHeaderTest, TestDocSet, SpellingQueryConverterTest, TestChildDocTransformer, TestComponentsName, TestStressRecovery, SuggesterTest, TestReload, TestConfig, TestFreeTextSuggestions, TriLevelCompositeIdRoutingTest, DeleteReplicaTest, TestJmxIntegration, PingRequestHandlerTest, TestTrie, BJQParserTest, TestNonNRTOpen, AddSchemaFieldsUpdateProcessorFactoryTest, DistributedSuggestComponentTest, OpenExchangeRatesOrgProviderTest, PreAnalyzedUpdateProcessorTest, DistributedExpandComponentTest, TestOrdValues, ReturnFieldsTest, CachingDirectoryFactoryTest]
   [junit4] Completed on J0 in 8.21s, 1 test, 1 error <<< FAILURES!

[...truncated 396 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestZkChroot
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.TestZkChroot-D4A7220426F8F346-001/init-core-data-001
   [junit4]   2> 1668461 T4989 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
   [junit4]   2> 1668464 T4989 oas.SolrTestCaseJ4.setUp ###Starting testInitPathExists
   [junit4]   2> 1668465 T4989 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1668466 T4990 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1668566 T4989 oasc.ZkTestServer.run start zk server on port:58832
   [junit4]   2> 1668566 T4989 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1668567 T4989 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1668569 T4996 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2d8dfb name:ZooKeeperConnection Watcher:127.0.0.1:58832 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1668569 T4989 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1668569 T4989 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1668570 T4989 oascc.SolrZkClient.makePath makePath: /foo/bar4
   [junit4]   2> 1668573 T4989 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/'
   [junit4]   2> 1668587 T4989 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/solr.xml
   [junit4]   2> 1668592 T4989 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr
   [junit4]   2> 1668593 T4989 oasc.CoreContainer.<init> New CoreContainer 12306384
   [junit4]   2> 1668593 T4989 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/]
   [junit4]   2> 1668594 T4989 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 1668594 T4989 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 1668595 T4989 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 1668595 T4989 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1668595 T4989 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1668596 T4989 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1668596 T4989 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1668596 T4989 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1668596 T4989 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1668597 T4989 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=0&connTimeout=0&retry=false
   [junit4]   2> 1668597 T4989 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1668598 T4989 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1668598 T4989 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 1668598 T4989 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58832/foo/bar4
   [junit4]   2> 1668599 T4989 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1668599 T4989 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1668600 T4989 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1668600 T4998 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a5cba name:ZooKeeperConnection Watcher:127.0.0.1:58832 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1668601 T4989 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1668601 T4989 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1668604 T4989 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1668605 T5000 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19117e6 name:ZooKeeperConnection Watcher:127.0.0.1:58832/foo/bar4 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1668605 T4989 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1668606 T4989 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1668609 T4989 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1668611 T4989 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1668613 T4989 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1668616 T4989 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1668618 T4989 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1668619 T4989 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/188.138.97.18:8983_solr
   [junit4]   2> 1668620 T4989 oascc.SolrZkClient.makePath makePath: /live_nodes/188.138.97.18:8983_solr
   [junit4]   2> 1668622 T4989 oascc.SolrZkClient.makePath makePath: /collections
   [junit4]   2> 1668623 T4989 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1668625 T4989 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1668627 T4989 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1668629 T4989 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 188.138.97.18:8983_solr
   [junit4]   2> 1668629 T4989 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1668630 T4989 oasc.Overseer.start Overseer (id=92453848074551298-188.138.97.18:8983_solr-n_0000000000) starting
   [junit4]   2> 1668632 T4989 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1668636 T4989 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1668637 T5002 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1668637 T4989 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1668639 T4989 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1668640 T4989 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1668642 T4989 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr
   [junit4]   2> 1668642 T5001 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1668646 T4989 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/
   [junit4]   2> 1668647 T4989 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1668647 T4989 oasc.ZkController.bootstrapConf bootstrapping config for 1 cores into ZooKeeper using solr.xml from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/
   [junit4]   2> 1668647 T4989 oasc.ZkController.bootstrapConf Uploading directory /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/conf with name collection1 for SolrCore collection1
   [junit4]   2> 1668648 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/spellings.txt
   [junit4]   2> 1668650 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/stopwords.txt
   [junit4]   2> 1668652 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/scripts.conf
   [junit4]   2> 1668653 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/admin-extra.html
   [junit4]   2> 1668655 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/schema.xml
   [junit4]   2> 1668656 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/solrconfig.xml
   [junit4]   2> 1668658 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/admin-extra.menu-top.html
   [junit4]   2> 1668659 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/elevate.xml
   [junit4]   2> 1668661 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/_schema_analysis_synonyms_english.json
   [junit4]   2> 1668663 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_nl.txt
   [junit4]   2> 1668665 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_eu.txt
   [junit4]   2> 1668667 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ga.txt
   [junit4]   2> 1668669 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_fr.txt
   [junit4]   2> 1668670 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_bg.txt
   [junit4]   2> 1668672 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stemdict_nl.txt
   [junit4]   2> 1668674 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/contractions_ca.txt
   [junit4]   2> 1668675 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ja.txt
   [junit4]   2> 1668677 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_id.txt
   [junit4]   2> 1668679 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_it.txt
   [junit4]   2> 1668680 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_fi.txt
   [junit4]   2> 1668682 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/contractions_fr.txt
   [junit4]   2> 1668684 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_el.txt
   [junit4]   2> 1668686 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_en.txt
   [junit4]   2> 1668688 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_da.txt
   [junit4]   2> 1668689 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stoptags_ja.txt
   [junit4]   2> 1668691 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_no.txt
   [junit4]   2> 1668692 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_pt.txt
   [junit4]   2> 1668694 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/hyphenations_ga.txt
   [junit4]   2> 1668695 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ru.txt
   [junit4]   2> 1668696 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ro.txt
   [junit4]   2> 1668697 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_cz.txt
   [junit4]   2> 1668699 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_hu.txt
   [junit4]   2> 1668700 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_th.txt
   [junit4]   2> 1668701 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ckb.txt
   [junit4]   2> 1668702 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_gl.txt
   [junit4]   2> 1668703 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ar.txt
   [junit4]   2> 1668705 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_tr.txt
   [junit4]   2> 1668706 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_es.txt
   [junit4]   2> 1668707 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/userdict_ja.txt
   [junit4]   2> 1668708 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/contractions_it.txt
   [junit4]   2> 1668710 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ca.txt
   [junit4]   2> 1668711 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_lv.txt
   [junit4]   2> 1668712 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_de.txt
   [junit4]   2> 1668714 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/contractions_ga.txt
   [junit4]   2> 1668715 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_hi.txt
   [junit4]   2> 1668717 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_hy.txt
   [junit4]   2> 1668718 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_sv.txt
   [junit4]   2> 1668720 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_fa.txt
   [junit4]   2> 1668722 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/synonyms.txt
   [junit4]   2> 1668723 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/update-script.js
   [junit4]   2> 1668725 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facet_pivot.vm
   [junit4]   2> 1668728 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/jquery.autocomplete.css
   [junit4]   2> 1668730 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/debug.vm
   [junit4]   2> 1668733 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/query_form.vm
   [junit4]   2> 1668737 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/pagination_top.vm
   [junit4]   2> 1668741 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/did_you_mean.vm
   [junit4]   2> 1668744 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/query_spatial.vm
   [junit4]   2> 1668747 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/hit_plain.vm
   [junit4]   2> 1668749 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/pagination_bottom.vm
   [junit4]   2> 1668751 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/header.vm
   [junit4]   2> 1668753 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/layout.vm
   [junit4]   2> 1668754 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/suggest.vm
   [junit4]   2> 1668755 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/README.txt
   [junit4]   2> 1668757 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/head.vm
   [junit4]   2> 1668758 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/footer.vm
   [junit4]   2> 1668759 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/results_list.vm
   [junit4]   2> 1668760 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/jquery.autocomplete.js
   [junit4]   2> 1668762 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/hit_grouped.vm
   [junit4]   2> 1668763 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facet_fields.vm
   [junit4]   2> 1668765 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/query_group.vm
   [junit4]   2> 1668766 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/query.vm
   [junit4]   2> 1668772 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/cluster_results.vm
   [junit4]   2> 1668774 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/VM_global_library.vm
   [junit4]   2> 1668775 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facet_queries.vm
   [junit4]   2> 1668776 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/mime_type_lists.vm
   [junit4]   2> 1668777 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/error.vm
   [junit4]   2> 1668779 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/product_doc.vm
   [junit4]   2> 1668780 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/main.css
   [junit4]   2> 1668781 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/browse.vm
   [junit4]   2> 1668783 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/richtext_doc.vm
   [junit4]   2> 1668784 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/cluster.vm
   [junit4]   2> 1668785 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facets.vm
   [junit4]   2> 1668787 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/tabs.vm
   [junit4]   2> 1668789 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/join_doc.vm
   [junit4]   2> 1668790 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/hit.vm
   [junit4]   2> 1668791 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facet_ranges.vm
   [junit4]   2> 1668793 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1668794 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/_schema_analysis_stopwords_english.json
   [junit4]   2> 1668795 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/protwords.txt
   [junit4]   2> 1668797 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/_rest_managed.json
   [junit4]   2> 1668798 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/example.xsl
   [junit4]   2> 1668800 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/example_rss.xsl
   [junit4]   2> 1668802 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/updateXml.xsl
   [junit4]   2> 1668804 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/luke.xsl
   [junit4]   2> 1668806 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/example_atom.xsl
   [junit4]   2> 1668808 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/mapping-FoldToASCII.txt
   [junit4]   2> 1668810 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/clustering/carrot2/stc-attributes.xml
   [junit4]   2> 1668813 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/clustering/carrot2/lingo-attributes.xml
   [junit4]   2> 1668815 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/clustering/carrot2/kmeans-attributes.xml
   [junit4]   2> 1668818 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/admin-extra.menu-bottom.html
   [junit4]   2> 1668821 T4989 oascc.SolrZkClient.makePath makePath: /configs/collection1/currency.xml
   [junit4]   2> 1668824 T4989 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr
   [junit4]   2> 1668827 T4989 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/
   [junit4]   2> 1668828 T4989 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1668829 T5004 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1668829 T5004 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1668830 T5004 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1668830 T5005 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1668833 T5001 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://188.138.97.18:8983/solr",
   [junit4]   2> 	  "node_name":"188.138.97.18:8983_solr",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1668834 T5001 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1668837 T5005 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1669831 T5004 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1669831 T5004 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1669832 T5004 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:collection1
   [junit4]   2> 1669833 T5004 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1669835 T5004 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1669837 T5004 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=collection1 specified config exists in ZooKeeper
   [junit4]   2> 1669837 T5004 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/'
   [junit4]   2> 1669858 T5004 oasc.SolrConfig.initLibs Adding specified lib dirs to ClassLoader
   [junit4]   2> 1669860 T5004 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/extraction/lib (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/extraction/lib).
   [junit4]   2> 1669861 T5004 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 1669862 T5004 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/clustering/lib/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/clustering/lib).
   [junit4]   2> 1669862 T5004 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 1669863 T5004 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/langid/lib/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/langid/lib).
   [junit4]   2> 1669864 T5004 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 1669864 T5004 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/velocity/lib (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/velocity/lib).
   [junit4]   2> 1669865 T5004 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 1669884 T5004 oasu.SolrIndexConfig.<init> IndexWriter infoStream solr logging is enabled
   [junit4]   2> 1669886 T5004 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1669917 T5004 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1669919 T5004 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1669923 T5004 oass.IndexSchema.readSchema [collection1] Schema name=example
   [junit4]   2> 1670123 T5004 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1670124 T5004 oasr.RestManager$Registry.registerManagedResource Registered ManagedResource impl org.apache.solr.rest.schema.analysis.ManagedWordSetResource for path /schema/analysis/stopwords/english
   [junit4]   2> 1670124 T5004 oasr.RestManager$Registry.registerManagedResource Registered ManagedResource impl org.apache.solr.rest.schema.analysis.ManagedSynonymFilterFactory$SynonymManager for path /schema/analysis/synonyms/english
   [junit4]   2> 1670134 T5004 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1670137 T5004 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1670240 T5004 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1670240 T5004 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1670240 T5004 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data/
   [junit4]   2> 1670240 T5004 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e4099a
   [junit4]   2> 1670249 T5004 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for newSearcher: org.apache.solr.core.QuerySenderListener{queries=[]}
   [junit4]   2> 1670249 T5004 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for firstSearcher: org.apache.solr.core.QuerySenderListener{queries=[{q=static firstSearcher warming in solrconfig.xml}]}
   [junit4]   2> 1670250 T5004 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@1213f81
   [junit4]   2> 1670250 T5004 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 1670250 T5004 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1670251 T5004 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1670251 T5004 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 0 directories currently being tracked
   [junit4]   2> 1670251 T5004 oasc.CoreContainer.create ERROR Error creating core [collection1]: access denied ("java.io.FilePermission" "/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data" "write") org.apache.solr.common.SolrException: access denied ("java.io.FilePermission" "/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data" "write")
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:891)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.security.Acc

[...truncated too long message...]

tOptimizationTest, TestBadConfig, AnalyticsMergeStrategyTest, TestModifyConfFiles, TestSolrQueryParser, SpatialFilterTest, SliceStateTest, FunctionTest, RequestHandlersTest, TestCollapseQParserPlugin, VMParamsZkACLAndCredentialsProvidersTest, PolyFieldTest, TestReversedWildcardFilterFactory, TestRandomFaceting, TestQueryTypes, SearchHandlerTest, TestCollectionAPI, TestCoreContainer, DistanceFunctionTest, OverriddenZkACLAndCredentialsProvidersTest, TestQueryUtils, QueryFacetTest, TestIBSimilarityFactory, AutoCommitTest, MergeStrategyTest, TestAddFieldRealTimeGet, TestBlendedInfixSuggestions, TestSortingResponseWriter, TestPostingsSolrHighlighter, SolrPluginUtilsTest, TestSolrQueryParserDefaultOperatorResource, LoggingHandlerTest, DistributedQueryElevationComponentTest, ChaosMonkeySafeLeaderTest, DistributedFacetPivotSmallTest, TestInitQParser, TestHighlightDedupGrouping, TestClassNameShortening, FullSolrCloudDistribCmdsTest, LeaderElectionTest, ZkSolrClientTest, TestZkChroot]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestZkChroot -Dtests.seed=D4A7220426F8F346 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=mt_MT -Dtests.timezone=Turkey -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | TestZkChroot (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=5004, name=coreLoadExecutor-2689-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
   [junit4]    > 	at java.lang.Thread.getStackTrace(Thread.java:1552)
   [junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
   [junit4]    > 	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
   [junit4]    > 	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
   [junit4]    > 	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
   [junit4]    > 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
   [junit4]    > 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
   [junit4]    > 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
   [junit4]    > 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@c2af4e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@13773d)
   [junit4]    > 	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
   [junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
   [junit4]    > 	... 1 moreThrowable #2: com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=5020, name=coreLoadExecutor-2696-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
   [junit4]    > 	at java.lang.Thread.getStackTrace(Thread.java:1552)
   [junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
   [junit4]    > 	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
   [junit4]    > 	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
   [junit4]    > 	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
   [junit4]    > 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
   [junit4]    > 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
   [junit4]    > 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
   [junit4]    > 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@1ce4d3a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@13a71f6)
   [junit4]    > 	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
   [junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
   [junit4]    > 	... 1 moreThrowable #3: com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=5049, name=coreLoadExecutor-2706-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
   [junit4]    > 	at java.lang.Thread.getStackTrace(Thread.java:1552)
   [junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
   [junit4]    > 	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
   [junit4]    > 	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
   [junit4]    > 	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
   [junit4]    > 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
   [junit4]    > 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
   [junit4]    > 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
   [junit4]    > 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@114eb46 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@fa37bf)
   [junit4]    > 	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
   [junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
   [junit4]    > 	... 1 more
   [junit4] Completed on J1 in 12.96s, 4 tests, 3 errors <<< FAILURES!

[...truncated 659 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:471: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:60: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1358: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:961: There were test failures: 438 suites, 1830 tests, 3 suite-level errors, 2 errors, 45 ignored (23 assumptions)

Total time: 88 minutes 59 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.9.0-ea-b28 -server -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



RE: [JENKINS] Lucene-Solr-trunk-Linux (64bit/ibm-j9-jdk7) - Build # 11246 - Still Failing!

Posted by Uwe Schindler <uw...@thetaphi.de>.
The CachingLockFactory problem is caused by the change to FSLockFactory to actually create the lock directory and not rely on FSDirectory to do this before. This test did nothing before, but now it creates locks. I will change it to use NoLockFactory.

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: Policeman Jenkins Server [mailto:jenkins@thetaphi.de]
> Sent: Sunday, September 14, 2014 9:21 PM
> To: rmuir@apache.org; dev@lucene.apache.org
> Subject: [JENKINS] Lucene-Solr-trunk-Linux (64bit/ibm-j9-jdk7) - Build #
> 11246 - Still Failing!
> 
> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11246/
> Java: 64bit/ibm-j9-jdk7 -
> Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/u
> til/fst/FST;}
> 
> 5 tests failed.
> FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestZkChroot
> 
> Error Message:
> Resource in scope SUITE failed to close. Resource was registered from thread
> Thread[id=1831, name=coreLoadExecutor-1254-thread-1, state=RUNNABLE,
> group=TGRP-TestZkChroot], registration stack trace below.
> 
> Stack Trace:
> com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in
> scope SUITE failed to close. Resource was registered from thread
> Thread[id=1831, name=coreLoadExecutor-1254-thread-1, state=RUNNABLE,
> group=TGRP-TestZkChroot], registration stack trace below.
> 	at java.lang.Thread.getStackTraceImpl(Native Method)
> 	at java.lang.Thread.getStackTrace(Thread.java:1203)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(Rando
> mizedContext.java:166)
> 	at
> org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java
> :731)
> 	at
> org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:
> 1274)
> 	at
> org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1
> 165)
> 	at
> org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1
> 157)
> 	at
> org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.jav
> a:40)
> 	at
> org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.ja
> va:351)
> 	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
> 	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
> 	at
> org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
> 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
> 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:273)
> 	at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> a:1176)
> 	at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> va:641)
> 	at java.lang.Thread.run(Thread.java:853)
> Caused by: java.lang.AssertionError: Directory not closed:
> MockDirectoryWrapper(RAMDirectory@d8b83b0a
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6d50aeb8)
> 	at org.junit.Assert.fail(Assert.java:93)
> 	at
> org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(Rando
> mizedRunner.java:699)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(Rando
> mizedRunner.java:696)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(R
> andomizedContext.java:183)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(Ran
> domizedRunner.java:712)
> 	at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> ementAdapter.java:43)
> 	at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.
> run(ThreadLeakControl.java:365)
> 	... 1 more
> 
> 
> FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestZkChroot
> 
> Error Message:
> Resource in scope SUITE failed to close. Resource was registered from thread
> Thread[id=1849, name=coreLoadExecutor-1261-thread-1, state=RUNNABLE,
> group=TGRP-TestZkChroot], registration stack trace below.
> 
> Stack Trace:
> com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in
> scope SUITE failed to close. Resource was registered from thread
> Thread[id=1849, name=coreLoadExecutor-1261-thread-1, state=RUNNABLE,
> group=TGRP-TestZkChroot], registration stack trace below.
> 	at java.lang.Thread.getStackTraceImpl(Native Method)
> 	at java.lang.Thread.getStackTrace(Thread.java:1203)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(Rando
> mizedContext.java:166)
> 	at
> org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java
> :731)
> 	at
> org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:
> 1274)
> 	at
> org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1
> 165)
> 	at
> org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1
> 157)
> 	at
> org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.jav
> a:40)
> 	at
> org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.ja
> va:351)
> 	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
> 	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
> 	at
> org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
> 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
> 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:273)
> 	at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> a:1176)
> 	at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> va:641)
> 	at java.lang.Thread.run(Thread.java:853)
> Caused by: java.lang.AssertionError: Directory not closed:
> MockDirectoryWrapper(RAMDirectory@7a86ba59
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@d7401f91)
> 	at org.junit.Assert.fail(Assert.java:93)
> 	at
> org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(Rando
> mizedRunner.java:699)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(Rando
> mizedRunner.java:696)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(R
> andomizedContext.java:183)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(Ran
> domizedRunner.java:712)
> 	at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> ementAdapter.java:43)
> 	at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.
> run(ThreadLeakControl.java:365)
> 	... 1 more
> 
> 
> FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestZkChroot
> 
> Error Message:
> Resource in scope SUITE failed to close. Resource was registered from thread
> Thread[id=1874, name=coreLoadExecutor-1271-thread-1, state=RUNNABLE,
> group=TGRP-TestZkChroot], registration stack trace below.
> 
> Stack Trace:
> com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in
> scope SUITE failed to close. Resource was registered from thread
> Thread[id=1874, name=coreLoadExecutor-1271-thread-1, state=RUNNABLE,
> group=TGRP-TestZkChroot], registration stack trace below.
> 	at java.lang.Thread.getStackTraceImpl(Native Method)
> 	at java.lang.Thread.getStackTrace(Thread.java:1203)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(Rando
> mizedContext.java:166)
> 	at
> org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java
> :731)
> 	at
> org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:
> 1274)
> 	at
> org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1
> 165)
> 	at
> org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1
> 157)
> 	at
> org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.jav
> a:40)
> 	at
> org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.ja
> va:351)
> 	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
> 	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
> 	at
> org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
> 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
> 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:273)
> 	at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> a:1176)
> 	at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> va:641)
> 	at java.lang.Thread.run(Thread.java:853)
> Caused by: java.lang.AssertionError: Directory not closed:
> MockDirectoryWrapper(RAMDirectory@48c00581
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@de935c8b)
> 	at org.junit.Assert.fail(Assert.java:93)
> 	at
> org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(Rando
> mizedRunner.java:699)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(Rando
> mizedRunner.java:696)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(R
> andomizedContext.java:183)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(Ran
> domizedRunner.java:712)
> 	at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> ementAdapter.java:43)
> 	at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.
> run(ThreadLeakControl.java:365)
> 	... 1 more
> 
> 
> FAILED:  org.apache.solr.core.CachingDirectoryFactoryTest.stressTest
> 
> Error Message:
> Captured an uncaught exception in thread: Thread[id=43, name=Thread-14,
> state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
> 
> Stack Trace:
> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
> uncaught exception in thread: Thread[id=43, name=Thread-14,
> state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
> Caused by: java.security.AccessControlException: Access denied
> ("java.io.FilePermission" "path17" "write")
> 	at __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
> 	at
> java.security.AccessController.throwACE(AccessController.java:100)
> 	at
> java.security.AccessController.checkPermission(AccessController.java:174)
> 	at
> java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
> 	at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.j
> ava:388)
> 	at java.nio.file.Files.createDirectory(Files.java:639)
> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
> 	at java.nio.file.Files.createDirectories(Files.java:692)
> 	at
> org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
> 	at
> org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.ja
> va:81)
> 	at
> org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirec
> toryFactory.java:500)
> 	at
> org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.ja
> va:357)
> 	at
> org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(Caching
> DirectoryFactoryTest.java:190)
> 
> 
> FAILED:  org.apache.solr.core.RAMDirectoryFactoryTest.test
> 
> Error Message:
> Access denied ("java.io.FilePermission" "/fake/path" "write")
> 
> Stack Trace:
> java.security.AccessControlException: Access denied
> ("java.io.FilePermission" "/fake/path" "write")
> 	at
> __randomizedtesting.SeedInfo.seed([E298FB1A86585611:6ACCC4C028A43BE
> 9]:0)
> 	at
> java.security.AccessController.throwACE(AccessController.java:100)
> 	at
> java.security.AccessController.checkPermission(AccessController.java:174)
> 	at
> java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
> 	at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.j
> ava:388)
> 	at java.nio.file.Files.createDirectory(Files.java:639)
> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
> 	at java.nio.file.Files.createDirectories(Files.java:692)
> 	at
> org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
> 	at
> org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.ja
> va:81)
> 	at
> org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirec
> toryFactory.java:500)
> 	at
> org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.ja
> va:357)
> 	at
> org.apache.solr.core.RAMDirectoryFactoryTest.dotestOpenReturnsTheSame
> ForSamePath(RAMDirectoryFactoryTest.java:46)
> 	at
> org.apache.solr.core.RAMDirectoryFactoryTest.test(RAMDirectoryFactoryTe
> st.java:33)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> ava:94)
> 	at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> sorImpl.java:55)
> 	at java.lang.reflect.Method.invoke(Method.java:619)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(Randomize
> dRunner.java:1618)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(Rando
> mizedRunner.java:827)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(Rando
> mizedRunner.java:863)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(Rando
> mizedRunner.java:877)
> 	at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRule
> SetupTeardownChained.java:50)
> 	at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeA
> fterRule.java:46)
> 	at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1
> .evaluate(SystemPropertiesInvariantRule.java:55)
> 	at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleTh
> readAndTestName.java:49)
> 	at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRule
> IgnoreAfterMaxFailures.java:65)
> 	at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure
> .java:48)
> 	at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> ementAdapter.java:36)
> 	at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.
> run(ThreadLeakControl.java:365)
> 	at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask
> (ThreadLeakControl.java:798)
> 	at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadL
> eakControl.java:458)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(Ran
> domizedRunner.java:836)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(Rando
> mizedRunner.java:738)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(Rando
> mizedRunner.java:772)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(Rando
> mizedRunner.java:783)
> 	at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeA
> fterRule.java:46)
> 	at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreCl
> assName.java:42)
> 	at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1
> .evaluate(SystemPropertiesInvariantRule.java:55)
> 	at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMet
> hodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> 	at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMet
> hodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> 	at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> ementAdapter.java:36)
> 	at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> ementAdapter.java:36)
> 	at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> ementAdapter.java:36)
> 	at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAss
> ertionsRequired.java:43)
> 	at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure
> .java:48)
> 	at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRule
> IgnoreAfterMaxFailures.java:65)
> 	at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnore
> TestSuites.java:55)
> 	at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> ementAdapter.java:36)
> 	at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.
> run(ThreadLeakControl.java:365)
> 	at java.lang.Thread.run(Thread.java:853)
> 
> 
> 
> 
> Build Log:
> [...truncated 10655 lines...]
>    [junit4] Suite: org.apache.solr.core.CachingDirectoryFactoryTest
>    [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-
> trunk-Linux/solr/build/solr-
> core/test/J1/temp/solr.core.CachingDirectoryFactoryTest-
> E298FB1A86585611-001/init-core-data-001
>    [junit4]   2> 10076 T36 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl
> (true) and clientAuth (true)
>    [junit4]   2> 10082 T36 oas.SolrTestCaseJ4.setUp ###Starting stressTest
>    [junit4]   2> 10159 T43 oasc.CachingDirectoryFactory.injectLockFactory
> WARN No lockType configured for RAMDirectory@a096e6e2
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@59a3485c
> assuming 'simple'
>    [junit4]   2> 86 T43
> ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtExcepti
> on WARNING Uncaught exception in thread: Thread[Thread-14,5,TGRP-
> CachingDirectoryFactoryTest] java.security.AccessControlException: Access
> denied ("java.io.FilePermission" "path17" "write")
>    [junit4]   2> 	at
> __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
>    [junit4]   2> 	at
> java.security.AccessController.throwACE(AccessController.java:100)
>    [junit4]   2> 	at
> java.security.AccessController.checkPermission(AccessController.java:174)
>    [junit4]   2> 	at
> java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
>    [junit4]   2> 	at
> java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
>    [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
>    [junit4]   2> 	at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.j
> ava:388)
>    [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:639)
>    [junit4]   2> 	at
> java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
>    [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:692)
>    [junit4]   2> 	at
> org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
>    [junit4]   2> 	at
> org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.ja
> va:81)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirec
> toryFactory.java:500)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.ja
> va:357)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(Caching
> DirectoryFactoryTest.java:190)
>    [junit4]   2>
>    [junit4]   2> 10163 T40 oasc.CachingDirectoryFactory.injectLockFactory
> WARN No lockType configured for RAMDirectory@baae1c26
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@ae4a3824
> assuming 'simple'
>    [junit4]   2> 10169 T39 oasc.CachingDirectoryFactory.injectLockFactory
> WARN No lockType configured for RAMDirectory@944f9e9
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4015e736
> assuming 'simple'
>    [junit4]   2> 93 T40
> ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtExcepti
> on WARNING Uncaught exception in thread: Thread[Thread-11,5,TGRP-
> CachingDirectoryFactoryTest] java.security.AccessControlException: Access
> denied ("java.io.FilePermission" "path17" "write")
>    [junit4]   2> 	at
> __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
>    [junit4]   2> 	at
> java.security.AccessController.throwACE(AccessController.java:100)
>    [junit4]   2> 	at
> java.security.AccessController.checkPermission(AccessController.java:174)
>    [junit4]   2> 	at
> java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
>    [junit4]   2> 	at
> java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
>    [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
>    [junit4]   2> 	at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.j
> ava:388)
>    [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:639)
>    [junit4]   2> 	at
> java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
>    [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:692)
>    [junit4]   2> 	at
> org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
>    [junit4]   2> 	at
> org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.ja
> va:81)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirec
> toryFactory.java:500)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.ja
> va:357)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(Caching
> DirectoryFactoryTest.java:190)
>    [junit4]   2>
>    [junit4]   2> 10170 T46 oasc.CachingDirectoryFactory.injectLockFactory
> WARN No lockType configured for RAMDirectory@cc8c48d1
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@46dc4ef9
> assuming 'simple'
>    [junit4]   2> 10172 T38 oasc.CachingDirectoryFactory.injectLockFactory
> WARN No lockType configured for RAMDirectory@79cd1e49
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@d8b01404
> assuming 'simple'
>    [junit4]   2> 96 T46
> ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtExcepti
> on WARNING Uncaught exception in thread: Thread[Thread-17,5,TGRP-
> CachingDirectoryFactoryTest] java.security.AccessControlException: Access
> denied ("java.io.FilePermission" "path17" "write")
>    [junit4]   2> 	at
> __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
>    [junit4]   2> 	at
> java.security.AccessController.throwACE(AccessController.java:100)
>    [junit4]   2> 	at
> java.security.AccessController.checkPermission(AccessController.java:174)
>    [junit4]   2> 	at
> java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
>    [junit4]   2> 	at
> java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
>    [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
>    [junit4]   2> 	at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.j
> ava:388)
>    [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:639)
>    [junit4]   2> 	at
> java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
>    [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:692)
>    [junit4]   2> 	at
> org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
>    [junit4]   2> 	at
> org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.ja
> va:81)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirec
> toryFactory.java:500)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.ja
> va:357)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(Caching
> DirectoryFactoryTest.java:190)
>    [junit4]   2>
>    [junit4]   2> 10173 T44 oasc.CachingDirectoryFactory.injectLockFactory
> WARN No lockType configured for RAMDirectory@f3c9039c
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@9747e6f
> assuming 'simple'
>    [junit4]   2> 10175 T42 oasc.CachingDirectoryFactory.injectLockFactory
> WARN No lockType configured for RAMDirectory@a59d92ee
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4ba92936
> assuming 'simple'
>    [junit4]   2> 94 T39
> ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtExcepti
> on WARNING Uncaught exception in thread: Thread[Thread-10,5,TGRP-
> CachingDirectoryFactoryTest] java.security.AccessControlException: Access
> denied ("java.io.FilePermission" "path17" "write")
>    [junit4]   2> 	at
> __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
>    [junit4]   2> 	at
> java.security.AccessController.throwACE(AccessController.java:100)
>    [junit4]   2> 	at
> java.security.AccessController.checkPermission(AccessController.java:174)
>    [junit4]   2> 	at
> java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
>    [junit4]   2> 	at
> java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
>    [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
>    [junit4]   2> 	at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.j
> ava:388)
>    [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:639)
>    [junit4]   2> 	at
> java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
>    [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:692)
>    [junit4]   2> 	at
> org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
>    [junit4]   2> 	at
> org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.ja
> va:81)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirec
> toryFactory.java:500)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.ja
> va:357)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(Caching
> DirectoryFactoryTest.java:190)
>    [junit4]   2>
>    [junit4]   2> 99 T44
> ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtExcepti
> on WARNING Uncaught exception in thread: Thread[Thread-15,5,TGRP-
> CachingDirectoryFactoryTest] java.security.AccessControlException: Access
> denied ("java.io.FilePermission" "path17" "write")
>    [junit4]   2> 	at
> __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
>    [junit4]   2> 	at
> java.security.AccessController.throwACE(AccessController.java:100)
>    [junit4]   2> 	at
> java.security.AccessController.checkPermission(AccessController.java:174)
>    [junit4]   2> 	at
> java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
>    [junit4]   2> 	at
> java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
>    [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
>    [junit4]   2> 	at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.j
> ava:388)
>    [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:639)
>    [junit4]   2> 	at
> java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
>    [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:692)
>    [junit4]   2> 	at
> org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
>    [junit4]   2> 	at
> org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.ja
> va:81)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirec
> toryFactory.java:500)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.ja
> va:357)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(Caching
> DirectoryFactoryTest.java:190)
>    [junit4]   2>
>    [junit4]   2> 10176 T37 oasc.CachingDirectoryFactory.injectLockFactory
> WARN No lockType configured for RAMDirectory@3e5c575c
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@9cb60008
> assuming 'simple'
>    [junit4]   2> 100 T42
> ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtExcepti
> on WARNING Uncaught exception in thread: Thread[Thread-13,5,TGRP-
> CachingDirectoryFactoryTest] java.security.AccessControlException: Access
> denied ("java.io.FilePermission" "path17" "write")
>    [junit4]   2> 	at
> __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
>    [junit4]   2> 	at
> java.security.AccessController.throwACE(AccessController.java:100)
>    [junit4]   2> 	at
> java.security.AccessController.checkPermission(AccessController.java:174)
>    [junit4]   2> 	at
> java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
>    [junit4]   2> 	at
> java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
>    [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
>    [junit4]   2> 	at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.j
> ava:388)
>    [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:639)
>    [junit4]   2> 	at
> java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
>    [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:692)
>    [junit4]   2> 	at
> org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
>    [junit4]   2> 	at
> org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.ja
> va:81)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirec
> toryFactory.java:500)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.ja
> va:357)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(Caching
> DirectoryFactoryTest.java:190)
>    [junit4]   2>
>    [junit4]   2> 10179 T41 oasc.CachingDirectoryFactory.injectLockFactory
> WARN No lockType configured for RAMDirectory@4b469a59
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@496d2317
> assuming 'simple'
>    [junit4]   2> 10182 T45 oasc.CachingDirectoryFactory.injectLockFactory
> WARN No lockType configured for RAMDirectory@2d225d69
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1435a94a
> assuming 'simple'
>    [junit4]   2> 10183 T47 oasc.CachingDirectoryFactory.injectLockFactory
> WARN No lockType configured for RAMDirectory@5800c3b0
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3b869729
> assuming 'simple'
>    [junit4]   2> 97 T38
> ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtExcepti
> on WARNING Uncaught exception in thread: Thread[Thread-9,5,TGRP-
> CachingDirectoryFactoryTest] java.security.AccessControlException: Access
> denied ("java.io.FilePermission" "path17" "write")
>    [junit4]   2> 	at
> __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
>    [junit4]   2> 	at
> java.security.AccessController.throwACE(AccessController.java:100)
>    [junit4]   2> 	at
> java.security.AccessController.checkPermission(AccessController.java:174)
>    [junit4]   2> 	at
> java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
>    [junit4]   2> 	at
> java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
>    [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
>    [junit4]   2> 	at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.j
> ava:388)
>    [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:639)
>    [junit4]   2> 	at
> java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
>    [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:692)
>    [junit4]   2> 	at
> org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
>    [junit4]   2> 	at
> org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.ja
> va:81)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirec
> toryFactory.java:500)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.ja
> va:357)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(Caching
> DirectoryFactoryTest.java:190)
>    [junit4]   2>
>    [junit4]   2> 107 T45
> ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtExcepti
> on WARNING Uncaught exception in thread: Thread[Thread-16,5,TGRP-
> CachingDirectoryFactoryTest] java.security.AccessControlException: Access
> denied ("java.io.FilePermission" "path17" "write")
>    [junit4]   2> 	at
> __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
>    [junit4]   2> 	at
> java.security.AccessController.throwACE(AccessController.java:100)
>    [junit4]   2> 	at
> java.security.AccessController.checkPermission(AccessController.java:174)
>    [junit4]   2> 	at
> java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
>    [junit4]   2> 	at
> java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
>    [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
>    [junit4]   2> 	at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.j
> ava:388)
>    [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:639)
>    [junit4]   2> 	at
> java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
>    [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:692)
>    [junit4]   2> 	at
> org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
>    [junit4]   2> 	at
> org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.ja
> va:81)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirec
> toryFactory.java:500)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.ja
> va:357)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(Caching
> DirectoryFactoryTest.java:190)
>    [junit4]   2>
>    [junit4]   2> 103 T37
> ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtExcepti
> on WARNING Uncaught exception in thread: Thread[Thread-8,5,TGRP-
> CachingDirectoryFactoryTest] java.security.AccessControlException: Access
> denied ("java.io.FilePermission" "path17" "write")
>    [junit4]   2> 	at
> __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
>    [junit4]   2> 	at
> java.security.AccessController.throwACE(AccessController.java:100)
>    [junit4]   2> 	at
> java.security.AccessController.checkPermission(AccessController.java:174)
>    [junit4]   2> 	at
> java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
>    [junit4]   2> 	at
> java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
>    [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
>    [junit4]   2> 	at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.j
> ava:388)
>    [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:639)
>    [junit4]   2> 	at
> java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
>    [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:692)
>    [junit4]   2> 	at
> org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
>    [junit4]   2> 	at
> org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.ja
> va:81)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirec
> toryFactory.java:500)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.ja
> va:357)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(Caching
> DirectoryFactoryTest.java:190)
>    [junit4]   2>
>    [junit4]   2> 106 T41
> ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtExcepti
> on WARNING Uncaught exception in thread: Thread[Thread-12,5,TGRP-
> CachingDirectoryFactoryTest] java.security.AccessControlException: Access
> denied ("java.io.FilePermission" "path17" "write")
>    [junit4]   2> 	at
> __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
>    [junit4]   2> 	at
> java.security.AccessController.throwACE(AccessController.java:100)
>    [junit4]   2> 	at
> java.security.AccessController.checkPermission(AccessController.java:174)
>    [junit4]   2> 	at
> java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
>    [junit4]   2> 	at
> java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
>    [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
>    [junit4]   2> 	at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.j
> ava:388)
>    [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:639)
>    [junit4]   2> 	at
> java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
>    [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:692)
>    [junit4]   2> 	at
> org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
>    [junit4]   2> 	at
> org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.ja
> va:81)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirec
> toryFactory.java:500)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.ja
> va:357)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(Caching
> DirectoryFactoryTest.java:190)
>    [junit4]   2>
>    [junit4]   2> 108 T47
> ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtExcepti
> on WARNING Uncaught exception in thread: Thread[Thread-18,5,TGRP-
> CachingDirectoryFactoryTest] java.security.AccessControlException: Access
> denied ("java.io.FilePermission" "path17" "write")
>    [junit4]   2> 	at
> __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
>    [junit4]   2> 	at
> java.security.AccessController.throwACE(AccessController.java:100)
>    [junit4]   2> 	at
> java.security.AccessController.checkPermission(AccessController.java:174)
>    [junit4]   2> 	at
> java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
>    [junit4]   2> 	at
> java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
>    [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
>    [junit4]   2> 	at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.j
> ava:388)
>    [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:639)
>    [junit4]   2> 	at
> java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
>    [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:692)
>    [junit4]   2> 	at
> org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
>    [junit4]   2> 	at
> org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.ja
> va:81)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirec
> toryFactory.java:500)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.ja
> va:357)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(Caching
> DirectoryFactoryTest.java:190)
>    [junit4]   2>
>    [junit4]   2> 18092 T54 oasc.CachingDirectoryFactory.close Closing
> RAMDirectoryFactory - 0 directories currently being tracked
>    [junit4]   2> 18236 T36 oas.SolrTestCaseJ4.tearDown ###Ending stressTest
>    [junit4]   2> NOTE: reproduce with: ant test  -
> Dtestcase=CachingDirectoryFactoryTest -Dtests.method=stressTest -
> Dtests.seed=E298FB1A86585611 -Dtests.multiplier=3 -Dtests.slow=true -
> Dtests.locale=hu -Dtests.timezone=Asia/Thimphu -Dtests.file.encoding=ISO-
> 8859-1
>    [junit4] ERROR   8.26s J1 | CachingDirectoryFactoryTest.stressTest <<<
>    [junit4]    > Throwable #1:
> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
> uncaught exception in thread: Thread[id=43, name=Thread-14,
> state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
>    [junit4]    > Caused by: java.security.AccessControlException: Access denied
> ("java.io.FilePermission" "path17" "write")
>    [junit4]    > 	at
> __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
>    [junit4]    > 	at
> java.security.AccessController.throwACE(AccessController.java:100)
>    [junit4]    > 	at
> java.security.AccessController.checkPermission(AccessController.java:174)
>    [junit4]    > 	at
> java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
>    [junit4]    > 	at
> java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
>    [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
>    [junit4]    > 	at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.j
> ava:388)
>    [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:639)
>    [junit4]    > 	at
> java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
>    [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:692)
>    [junit4]    > 	at
> org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
>    [junit4]    > 	at
> org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.ja
> va:81)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirec
> toryFactory.java:500)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.ja
> va:357)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(Caching
> DirectoryFactoryTest.java:190)Throwable #2:
> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
> uncaught exception in thread: Thread[id=40, name=Thread-11,
> state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
>    [junit4]    > Caused by: java.security.AccessControlException: Access denied
> ("java.io.FilePermission" "path17" "write")
>    [junit4]    > 	at
> __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
>    [junit4]    > 	at
> java.security.AccessController.throwACE(AccessController.java:100)
>    [junit4]    > 	at
> java.security.AccessController.checkPermission(AccessController.java:174)
>    [junit4]    > 	at
> java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
>    [junit4]    > 	at
> java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
>    [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
>    [junit4]    > 	at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.j
> ava:388)
>    [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:639)
>    [junit4]    > 	at
> java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
>    [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:692)
>    [junit4]    > 	at
> org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
>    [junit4]    > 	at
> org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.ja
> va:81)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirec
> toryFactory.java:500)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.ja
> va:357)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(Caching
> DirectoryFactoryTest.java:190)Throwable #3:
> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
> uncaught exception in thread: Thread[id=39, name=Thread-10,
> state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
>    [junit4]    > Caused by: java.security.AccessControlException: Access denied
> ("java.io.FilePermission" "path17" "write")
>    [junit4]    > 	at
> __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
>    [junit4]    > 	at
> java.security.AccessController.throwACE(AccessController.java:100)
>    [junit4]    > 	at
> java.security.AccessController.checkPermission(AccessController.java:174)
>    [junit4]    > 	at
> java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
>    [junit4]    > 	at
> java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
>    [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
>    [junit4]    > 	at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.j
> ava:388)
>    [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:639)
>    [junit4]    > 	at
> java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
>    [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:692)
>    [junit4]    > 	at
> org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
>    [junit4]    > 	at
> org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.ja
> va:81)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirec
> toryFactory.java:500)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.ja
> va:357)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(Caching
> DirectoryFactoryTest.java:190)Throwable #4:
> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
> uncaught exception in thread: Thread[id=46, name=Thread-17,
> state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
>    [junit4]    > Caused by: java.security.AccessControlException: Access denied
> ("java.io.FilePermission" "path17" "write")
>    [junit4]    > 	at
> __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
>    [junit4]    > 	at
> java.security.AccessController.throwACE(AccessController.java:100)
>    [junit4]    > 	at
> java.security.AccessController.checkPermission(AccessController.java:174)
>    [junit4]    > 	at
> java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
>    [junit4]    > 	at
> java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
>    [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
>    [junit4]    > 	at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.j
> ava:388)
>    [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:639)
>    [junit4]    > 	at
> java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
>    [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:692)
>    [junit4]    > 	at
> org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
>    [junit4]    > 	at
> org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.ja
> va:81)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirec
> toryFactory.java:500)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.ja
> va:357)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(Caching
> DirectoryFactoryTest.java:190)Throwable #5:
> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
> uncaught exception in thread: Thread[id=38, name=Thread-9,
> state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
>    [junit4]    > Caused by: java.security.AccessControlException: Access denied
> ("java.io.FilePermission" "path17" "write")
>    [junit4]    > 	at
> __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
>    [junit4]    > 	at
> java.security.AccessController.throwACE(AccessController.java:100)
>    [junit4]    > 	at
> java.security.AccessController.checkPermission(AccessController.java:174)
>    [junit4]    > 	at
> java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
>    [junit4]    > 	at
> java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
>    [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
>    [junit4]    > 	at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.j
> ava:388)
>    [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:639)
>    [junit4]    > 	at
> java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
>    [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:692)
>    [junit4]    > 	at
> org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
>    [junit4]    > 	at
> org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.ja
> va:81)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirec
> toryFactory.java:500)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.ja
> va:357)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(Caching
> DirectoryFactoryTest.java:190)Throwable #6:
> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
> uncaught exception in thread: Thread[id=44, name=Thread-15,
> state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
>    [junit4]    > Caused by: java.security.AccessControlException: Access denied
> ("java.io.FilePermission" "path17" "write")
>    [junit4]    > 	at
> __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
>    [junit4]    > 	at
> java.security.AccessController.throwACE(AccessController.java:100)
>    [junit4]    > 	at
> java.security.AccessController.checkPermission(AccessController.java:174)
>    [junit4]    > 	at
> java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
>    [junit4]    > 	at
> java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
>    [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
>    [junit4]    > 	at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.j
> ava:388)
>    [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:639)
>    [junit4]    > 	at
> java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
>    [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:692)
>    [junit4]    > 	at
> org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
>    [junit4]    > 	at
> org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.ja
> va:81)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirec
> toryFactory.java:500)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.ja
> va:357)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(Caching
> DirectoryFactoryTest.java:190)Throwable #7:
> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
> uncaught exception in thread: Thread[id=42, name=Thread-13,
> state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
>    [junit4]    > Caused by: java.security.AccessControlException: Access denied
> ("java.io.FilePermission" "path17" "write")
>    [junit4]    > 	at
> __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
>    [junit4]    > 	at
> java.security.AccessController.throwACE(AccessController.java:100)
>    [junit4]    > 	at
> java.security.AccessController.checkPermission(AccessController.java:174)
>    [junit4]    > 	at
> java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
>    [junit4]    > 	at
> java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
>    [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
>    [junit4]    > 	at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.j
> ava:388)
>    [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:639)
>    [junit4]    > 	at
> java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
>    [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:692)
>    [junit4]    > 	at
> org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
>    [junit4]    > 	at
> org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.ja
> va:81)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirec
> toryFactory.java:500)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.ja
> va:357)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(Caching
> DirectoryFactoryTest.java:190)Throwable #8:
> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
> uncaught exception in thread: Thread[id=37, name=Thread-8,
> state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
>    [junit4]    > Caused by: java.security.AccessControlException: Access denied
> ("java.io.FilePermission" "path17" "write")
>    [junit4]    > 	at
> __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
>    [junit4]    > 	at
> java.security.AccessController.throwACE(AccessController.java:100)
>    [junit4]    > 	at
> java.security.AccessController.checkPermission(AccessController.java:174)
>    [junit4]    > 	at
> java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
>    [junit4]    > 	at
> java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
>    [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
>    [junit4]    > 	at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.j
> ava:388)
>    [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:639)
>    [junit4]    > 	at
> java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
>    [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:692)
>    [junit4]    > 	at
> org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
>    [junit4]    > 	at
> org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.ja
> va:81)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirec
> toryFactory.java:500)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.ja
> va:357)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(Caching
> DirectoryFactoryTest.java:190)Throwable #9:
> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
> uncaught exception in thread: Thread[id=41, name=Thread-12,
> state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
>    [junit4]    > Caused by: java.security.AccessControlException: Access denied
> ("java.io.FilePermission" "path17" "write")
>    [junit4]    > 	at
> __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
>    [junit4]    > 	at
> java.security.AccessController.throwACE(AccessController.java:100)
>    [junit4]    > 	at
> java.security.AccessController.checkPermission(AccessController.java:174)
>    [junit4]    > 	at
> java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
>    [junit4]    > 	at
> java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
>    [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
>    [junit4]    > 	at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.j
> ava:388)
>    [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:639)
>    [junit4]    > 	at
> java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
>    [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:692)
>    [junit4]    > 	at
> org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
>    [junit4]    > 	at
> org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.ja
> va:81)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirec
> toryFactory.java:500)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.ja
> va:357)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(Caching
> DirectoryFactoryTest.java:190)Throwable #10:
> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
> uncaught exception in thread: Thread[id=45, name=Thread-16,
> state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
>    [junit4]    > Caused by: java.security.AccessControlException: Access denied
> ("java.io.FilePermission" "path17" "write")
>    [junit4]    > 	at
> __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
>    [junit4]    > 	at
> java.security.AccessController.throwACE(AccessController.java:100)
>    [junit4]    > 	at
> java.security.AccessController.checkPermission(AccessController.java:174)
>    [junit4]    > 	at
> java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
>    [junit4]    > 	at
> java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
>    [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
>    [junit4]    > 	at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.j
> ava:388)
>    [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:639)
>    [junit4]    > 	at
> java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
>    [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:692)
>    [junit4]    > 	at
> org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
>    [junit4]    > 	at
> org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.ja
> va:81)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirec
> toryFactory.java:500)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.ja
> va:357)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(Caching
> DirectoryFactoryTest.java:190)Throwable #11:
> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
> uncaught exception in thread: Thread[id=47, name=Thread-18,
> state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
>    [junit4]    > Caused by: java.security.AccessControlException: Access denied
> ("java.io.FilePermission" "path17" "write")
>    [junit4]    > 	at
> __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
>    [junit4]    > 	at
> java.security.AccessController.throwACE(AccessController.java:100)
>    [junit4]    > 	at
> java.security.AccessController.checkPermission(AccessController.java:174)
>    [junit4]    > 	at
> java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
>    [junit4]    > 	at
> java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
>    [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
>    [junit4]    > 	at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.j
> ava:388)
>    [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:639)
>    [junit4]    > 	at
> java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
>    [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:692)
>    [junit4]    > 	at
> org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
>    [junit4]    > 	at
> org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.ja
> va:81)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirec
> toryFactory.java:500)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.ja
> va:357)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(Caching
> DirectoryFactoryTest.java:190)
>    [junit4]   2> 18333 T36 oas.SolrTestCaseJ4.deleteCore ###deleteCore
>    [junit4]   2> NOTE: test params are:
> codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=Co
> mpressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION,
> chunkSize=318),
> termVectorsFormat=CompressingTermVectorsFormat(compressionMode=F
> AST_DECOMPRESSION, chunkSize=318)),
> sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=hu,
> timezone=Asia/Thimphu
>    [junit4]   2> NOTE: Linux 3.13.0-35-generic amd64/IBM Corporation 1.7.0
> (64-bit)/cpus=8,threads=1,free=7770776,total=35061760
>    [junit4]   2> NOTE: All tests run in this JVM: [OutputWriterTest,
> SpatialFilterTest, CachingDirectoryFactoryTest]
>    [junit4] Completed on J1 in 8.38s, 1 test, 1 error <<< FAILURES!
> 
> [...truncated 113 lines...]
>    [junit4] Suite: org.apache.solr.core.RAMDirectoryFactoryTest
>    [junit4]   2> 95703 T322 oasc.CachingDirectoryFactory.injectLockFactory
> WARN No lockType configured for RAMDirectory@ff26adfe
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@f6628f14
> assuming 'simple'
>    [junit4]   2> NOTE: reproduce with: ant test  -
> Dtestcase=RAMDirectoryFactoryTest -Dtests.method=test -
> Dtests.seed=E298FB1A86585611 -Dtests.multiplier=3 -Dtests.slow=true -
> Dtests.locale=sh_CS -Dtests.timezone=Africa/Lagos -
> Dtests.file.encoding=ISO-8859-1
>    [junit4] ERROR   0.01s J1 | RAMDirectoryFactoryTest.test <<<
>    [junit4]    > Throwable #1: java.security.AccessControlException: Access
> denied ("java.io.FilePermission" "/fake/path" "write")
>    [junit4]    > 	at
> __randomizedtesting.SeedInfo.seed([E298FB1A86585611:6ACCC4C028A43BE
> 9]:0)
>    [junit4]    > 	at
> java.security.AccessController.throwACE(AccessController.java:100)
>    [junit4]    > 	at
> java.security.AccessController.checkPermission(AccessController.java:174)
>    [junit4]    > 	at
> java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
>    [junit4]    > 	at
> java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
>    [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
>    [junit4]    > 	at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.j
> ava:388)
>    [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:639)
>    [junit4]    > 	at
> java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
>    [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:692)
>    [junit4]    > 	at
> org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
>    [junit4]    > 	at
> org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.ja
> va:81)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirec
> toryFactory.java:500)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.ja
> va:357)
>    [junit4]    > 	at
> org.apache.solr.core.RAMDirectoryFactoryTest.dotestOpenReturnsTheSame
> ForSamePath(RAMDirectoryFactoryTest.java:46)
>    [junit4]    > 	at
> org.apache.solr.core.RAMDirectoryFactoryTest.test(RAMDirectoryFactoryTe
> st.java:33)
>    [junit4]    > 	at java.lang.Thread.run(Thread.java:853)
>    [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{},
> sim=DefaultSimilarity, locale=sh_CS, timezone=Africa/Lagos
>    [junit4]   2> NOTE: Linux 3.13.0-35-generic amd64/IBM Corporation 1.7.0
> (64-bit)/cpus=8,threads=1,free=24885040,total=85917696
>    [junit4]   2> NOTE: All tests run in this JVM: [OutputWriterTest,
> SpatialFilterTest, CachingDirectoryFactoryTest, HdfsDirectoryTest,
> TestCopyFieldCollectionResource, OpenCloseCoreStressTest,
> NumericFieldsTest, QueryElevationComponentTest,
> DistributedQueryElevationComponentTest, CacheHeaderTest,
> BasicDistributedZk2Test, TestReplicationHandlerBackup, SliceStateTest,
> AutoCommitTest, OverriddenZkACLAndCredentialsProvidersTest,
> CoreMergeIndexesAdminHandlerTest,
> UpdateRequestProcessorFactoryTest, LegacyHTMLStripCharFilterTest,
> ScriptEngineTest, RAMDirectoryFactoryTest]
>    [junit4] Completed on J1 in 0.03s, 1 test, 1 error <<< FAILURES!
> 
> [...truncated 579 lines...]
>    [junit4] Suite: org.apache.solr.cloud.TestZkChroot
>    [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-
> trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.TestZkChroot-
> E298FB1A86585611-001/init-core-data-001
>    [junit4]   2> 706204 T1816 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl
> (true) and clientAuth (true)
>    [junit4]   2> 706207 T1816 oas.SolrTestCaseJ4.setUp ###Starting
> testWithUploadDir
>    [junit4]   2> 706207 T1816 oasc.ZkTestServer.run STARTING ZK TEST SERVER
>    [junit4]   1> client port:0.0.0.0/0.0.0.0:0
>    [junit4]   2> 706208 T1817 oasc.ZkTestServer$ZKServerMain.runFromConfig
> Starting server
>    [junit4]   2> 706308 T1816 oasc.ZkTestServer.run start zk server on
> port:44091
>    [junit4]   2> 706309 T1816
> oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> ZkCredentialsProvider
>    [junit4]   2> 706310 T1816 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 706313 T1823 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@4e404da3
> name:ZooKeeperConnection Watcher:127.0.0.1:44091 got event
> WatchedEvent state:SyncConnected type:None path:null path:null
> type:None
>    [junit4]   2> 706313 T1816 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 706314 T1816 oascc.SolrZkClient.createZkACLProvider Using
> default ZkACLProvider
>    [junit4]   2> 706315 T1816 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-
> Solr-trunk-Linux/solr/example/solr/'
>    [junit4]   2> 706328 T1816 oasc.ConfigSolr.fromFile Loading container
> configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-
> Linux/solr/example/solr/solr.xml
>    [junit4]   2> 706359 T1816 oasc.CorePropertiesLocator.<init> Config-defined
> core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-
> Linux/solr/example/solr
>    [junit4]   2> 706360 T1816 oasc.CoreContainer.<init> New CoreContainer -
> 225327910
>    [junit4]   2> 706360 T1816 oasc.CoreContainer.load Loading cores into
> CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-
> trunk-Linux/solr/example/solr/]
>    [junit4]   2> 706361 T1816 oashc.HttpShardHandlerFactory.getParameter
> Setting socketTimeout to: 0
>    [junit4]   2> 706362 T1816 oashc.HttpShardHandlerFactory.getParameter
> Setting urlScheme to: null
>    [junit4]   2> 706362 T1816 oashc.HttpShardHandlerFactory.getParameter
> Setting connTimeout to: 0
>    [junit4]   2> 706362 T1816 oashc.HttpShardHandlerFactory.getParameter
> Setting maxConnectionsPerHost to: 20
>    [junit4]   2> 706363 T1816 oashc.HttpShardHandlerFactory.getParameter
> Setting corePoolSize to: 0
>    [junit4]   2> 706363 T1816 oashc.HttpShardHandlerFactory.getParameter
> Setting maximumPoolSize to: 2147483647
>    [junit4]   2> 706363 T1816 oashc.HttpShardHandlerFactory.getParameter
> Setting maxThreadIdleTime to: 5
>    [junit4]   2> 706364 T1816 oashc.HttpShardHandlerFactory.getParameter
> Setting sizeOfQueue to: -1
>    [junit4]   2> 706364 T1816 oashc.HttpShardHandlerFactory.getParameter
> Setting fairnessPolicy to: false
>    [junit4]   2> 706366 T1816 oasu.UpdateShardHandler.<init> Creating
> UpdateShardHandler HTTP client with params:
> socketTimeout=0&connTimeout=0&retry=false
>    [junit4]   2> 706367 T1816 oasl.LogWatcher.createWatcher SLF4J impl is
> org.slf4j.impl.Log4jLoggerFactory
>    [junit4]   2> 706367 T1816 oasl.LogWatcher.newRegisteredLogWatcher
> Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>    [junit4]   2> 706367 T1816 oasc.CoreContainer.load Host Name:
>    [junit4]   2> 706368 T1816 oasc.ZkContainer.initZooKeeper Zookeeper
> client=127.0.0.1:44091/foo/bar3
>    [junit4]   2> 706368 T1816 oasc.ZkController.checkChrootPath zkHost
> includes chroot
>    [junit4]   2> 706368 T1816
> oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> ZkCredentialsProvider
>    [junit4]   2> 706369 T1816 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 706370 T1825 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@6ff14d4d
> name:ZooKeeperConnection Watcher:127.0.0.1:44091 got event
> WatchedEvent state:SyncConnected type:None path:null path:null
> type:None
>    [junit4]   2> 706371 T1816 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 706371 T1816 oascc.SolrZkClient.createZkACLProvider Using
> default ZkACLProvider
>    [junit4]   2> 706372 T1816 oascc.SolrZkClient.makePath makePath:
> /foo/bar3
>    [junit4]   2> 706377 T1816 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 706378 T1827 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@bbbbf8aa
> name:ZooKeeperConnection Watcher:127.0.0.1:44091/foo/bar3 got event
> WatchedEvent state:SyncConnected type:None path:null path:null
> type:None
>    [junit4]   2> 706378 T1816 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 706379 T1816 oascc.SolrZkClient.makePath makePath:
> /overseer/queue
>    [junit4]   2> 706383 T1816 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-queue-work
>    [junit4]   2> 706386 T1816 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-running
>    [junit4]   2> 706389 T1816 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-completed
>    [junit4]   2> 706392 T1816 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-failure
>    [junit4]   2> 706395 T1816 oascc.SolrZkClient.makePath makePath:
> /live_nodes
>    [junit4]   2> 706397 T1816 oasc.ZkController.createEphemeralLiveNode
> Register node as live in ZooKeeper:/live_nodes/188.138.97.18:8983_solr
>    [junit4]   2> 706398 T1816 oascc.SolrZkClient.makePath makePath:
> /live_nodes/188.138.97.18:8983_solr
>    [junit4]   2> 706400 T1816 oascc.SolrZkClient.makePath makePath:
> /collections
>    [junit4]   2> 706402 T1816 oascc.SolrZkClient.makePath makePath:
> /overseer_elect
>    [junit4]   2> 706404 T1816 oascc.SolrZkClient.makePath makePath:
> /overseer_elect/election
>    [junit4]   2> 706406 T1816 oasc.Overseer.close Overseer (id=null) closing
>    [junit4]   2> 706408 T1816 oasc.OverseerElectionContext.runLeaderProcess I
> am going to be the leader 188.138.97.18:8983_solr
>    [junit4]   2> 706408 T1816 oascc.SolrZkClient.makePath makePath:
> /overseer_elect/leader
>    [junit4]   2> 706410 T1816 oasc.Overseer.start Overseer
> (id=92453081980010498-188.138.97.18:8983_solr-n_0000000000) starting
>    [junit4]   2> 706413 T1816 oascc.SolrZkClient.makePath makePath:
> /overseer/queue-work
>    [junit4]   2> 706423 T1816 oasc.OverseerAutoReplicaFailoverThread.<init>
> Starting OverseerAutoReplicaFailoverThread
> autoReplicaFailoverWorkLoopDelay=10000
> autoReplicaFailoverWaitAfterExpiration=30000
> autoReplicaFailoverBadNodeExpiration=60000
>    [junit4]   2> 706425 T1829 oasc.OverseerCollectionProcessor.run Process
> current queue of collection creations
>    [junit4]   2> 706425 T1816 oascc.SolrZkClient.makePath makePath:
> /clusterstate.json
>    [junit4]   2> 706429 T1816 oascc.SolrZkClient.makePath makePath:
> /aliases.json
>    [junit4]   2> 706430 T1816
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating
> cluster state from ZooKeeper...
>    [junit4]   2> 706433 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/spellings.txt
>    [junit4]   2> 706433 T1828 oasc.Overseer$ClusterStateUpdater.run Starting
> to work on the main queue
>    [junit4]   2> 706435 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/stopwords.txt
>    [junit4]   2> 706438 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/scripts.conf
>    [junit4]   2> 706439 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/admin-extra.html
>    [junit4]   2> 706441 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/schema.xml
>    [junit4]   2> 706443 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/solrconfig.xml
>    [junit4]   2> 706446 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/admin-extra.menu-top.html
>    [junit4]   2> 706448 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/elevate.xml
>    [junit4]   2> 706451 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/_schema_analysis_synonyms_english.json
>    [junit4]   2> 706454 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/stopwords_nl.txt
>    [junit4]   2> 706457 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/stopwords_eu.txt
>    [junit4]   2> 706459 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/stopwords_ga.txt
>    [junit4]   2> 706462 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/stopwords_fr.txt
>    [junit4]   2> 706465 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/stopwords_bg.txt
>    [junit4]   2> 706468 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/stemdict_nl.txt
>    [junit4]   2> 706471 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/contractions_ca.txt
>    [junit4]   2> 706474 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/stopwords_ja.txt
>    [junit4]   2> 706477 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/stopwords_id.txt
>    [junit4]   2> 706481 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/stopwords_it.txt
>    [junit4]   2> 706485 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/stopwords_fi.txt
>    [junit4]   2> 706488 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/contractions_fr.txt
>    [junit4]   2> 706491 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/stopwords_el.txt
>    [junit4]   2> 706494 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/stopwords_en.txt
>    [junit4]   2> 706497 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/stopwords_da.txt
>    [junit4]   2> 706500 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/stoptags_ja.txt
>    [junit4]   2> 706503 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/stopwords_no.txt
>    [junit4]   2> 706505 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/stopwords_pt.txt
>    [junit4]   2> 706509 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/hyphenations_ga.txt
>    [junit4]   2> 706513 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/stopwords_ru.txt
>    [junit4]   2> 706517 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/stopwords_ro.txt
>    [junit4]   2> 706520 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/stopwords_cz.txt
>    [junit4]   2> 706524 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/stopwords_hu.txt
>    [junit4]   2> 706528 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/stopwords_th.txt
>    [junit4]   2> 706531 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/stopwords_ckb.txt
>    [junit4]   2> 706535 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/stopwords_gl.txt
>    [junit4]   2> 706538 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/stopwords_ar.txt
>    [junit4]   2> 706542 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/stopwords_tr.txt
>    [junit4]   2> 706545 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/stopwords_es.txt
>    [junit4]   2> 706549 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/userdict_ja.txt
>    [junit4]   2> 706552 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/contractions_it.txt
>    [junit4]   2> 706556 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/stopwords_ca.txt
>    [junit4]   2> 706559 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/stopwords_lv.txt
>    [junit4]   2> 706563 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/stopwords_de.txt
>    [junit4]   2> 706566 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/contractions_ga.txt
>    [junit4]   2> 706569 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/stopwords_hi.txt
>    [junit4]   2> 706572 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/stopwords_hy.txt
>    [junit4]   2> 706574 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/stopwords_sv.txt
>    [junit4]   2> 706577 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/lang/stopwords_fa.txt
>    [junit4]   2> 706579 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/synonyms.txt
>    [junit4]   2> 706582 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/update-script.js
>    [junit4]   2> 706584 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/facet_pivot.vm
>    [junit4]   2> 706587 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/jquery.autocomplete.css
>    [junit4]   2> 706591 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/debug.vm
>    [junit4]   2> 706594 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/query_form.vm
>    [junit4]   2> 706597 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/pagination_top.vm
>    [junit4]   2> 706601 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/did_you_mean.vm
>    [junit4]   2> 706604 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/query_spatial.vm
>    [junit4]   2> 706608 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/hit_plain.vm
>    [junit4]   2> 706611 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/pagination_bottom.vm
>    [junit4]   2> 706614 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/header.vm
>    [junit4]   2> 706618 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/layout.vm
>    [junit4]   2> 706622 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/suggest.vm
>    [junit4]   2> 706625 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/README.txt
>    [junit4]   2> 706629 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/head.vm
>    [junit4]   2> 706633 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/footer.vm
>    [junit4]   2> 706637 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/results_list.vm
>    [junit4]   2> 706640 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/jquery.autocomplete.js
>    [junit4]   2> 706644 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/hit_grouped.vm
>    [junit4]   2> 706647 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/facet_fields.vm
>    [junit4]   2> 706649 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/query_group.vm
>    [junit4]   2> 706650 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/query.vm
>    [junit4]   2> 706652 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/cluster_results.vm
>    [junit4]   2> 706654 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/VM_global_library.vm
>    [junit4]   2> 706656 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/facet_queries.vm
>    [junit4]   2> 706657 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/mime_type_lists.vm
>    [junit4]   2> 706659 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/error.vm
>    [junit4]   2> 706661 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/product_doc.vm
>    [junit4]   2> 706662 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/main.css
>    [junit4]   2> 706664 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/browse.vm
>    [junit4]   2> 706666 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/richtext_doc.vm
>    [junit4]   2> 706669 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/cluster.vm
>    [junit4]   2> 706673 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/facets.vm
>    [junit4]   2> 706677 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/tabs.vm
>    [junit4]   2> 706681 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/join_doc.vm
>    [junit4]   2> 706684 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/hit.vm
>    [junit4]   2> 706687 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/velocity/facet_ranges.vm
>    [junit4]   2> 706691 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/mapping-ISOLatin1Accent.txt
>    [junit4]   2> 706693 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/_schema_analysis_stopwords_english.json
>    [junit4]   2> 706696 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/protwords.txt
>    [junit4]   2> 706699 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/_rest_managed.json
>    [junit4]   2> 706701 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/xslt/example.xsl
>    [junit4]   2> 706703 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/xslt/example_rss.xsl
>    [junit4]   2> 706704 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/xslt/updateXml.xsl
>    [junit4]   2> 706706 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/xslt/luke.xsl
>    [junit4]   2> 706707 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/xslt/example_atom.xsl
>    [junit4]   2> 706709 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/mapping-FoldToASCII.txt
>    [junit4]   2> 706710 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/clustering/carrot2/stc-attributes.xml
>    [junit4]   2> 706713 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/clustering/carrot2/lingo-attributes.xml
>    [junit4]   2> 706716 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/clustering/carrot2/kmeans-attributes.xml
>    [junit4]   2> 706718 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/admin-extra.menu-bottom.html
>    [junit4]   2> 706720 T1816 oascc.SolrZkClient.makePath makePath:
> /configs/testWithUploadDir/currency.xml
>    [junit4]   2> 706724 T1816 oasc.CorePropertiesLocator.discover Looking for
> core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-
> trunk-Linux/solr/example/solr
>    [junit4]   2> 706727 T1816 oasc.CorePropertiesLocator.discoverUnder Found
> core collection1 in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-
> Linux/solr/example/solr/collection1/
>    [junit4]   2> 706727 T1816 oasc.CorePropertiesLocator.discover Found 1 core
> definitions
>    [junit4]   2> 706728 T1831 oasc.ZkController.publish publishing
> core=collection1 state=down collection=collection1
>    [junit4]   2> 706728 T1831 oasc.ZkController.publish numShards not found
> on descriptor - reading it from system property
>    [junit4]   2> 706730 T1831 oasc.ZkController.waitForCoreNodeName look for
> our core node name
>    [junit4]   2> 706730 T1832
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired
> on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 706733 T1828 oasc.Overseer$ClusterStateUpdater.updateState
> Update state numShards=null message={
>    [junit4]   2> 	  "base_url":"http://188.138.97.18:8983/solr",
>    [junit4]   2> 	  "roles":null,
>    [junit4]   2> 	  "collection":"collection1",
>    [junit4]   2> 	  "node_name":"188.138.97.18:8983_solr",
>    [junit4]   2> 	  "core":"collection1",
>    [junit4]   2> 	  "operation":"state",
>    [junit4]   2> 	  "shard":null,
>    [junit4]   2> 	  "state":"down"}
>    [junit4]   2> 706734 T1828 oasc.Overseer$ClusterStateUpdater.updateState
> Assigning new node to shard shard=shard1
>    [junit4]   2> 706738 T1832 oascc.ZkStateReader$2.process A cluster state
> change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>    [junit4]   2> 707730 T1831 oasc.ZkController.waitForShardId waiting to find
> shard id in clusterstate for collection1
>    [junit4]   2> 707731 T1831 oasc.ZkController.createCollectionZkNode Check
> for collection zkNode:collection1
>    [junit4]   2> 707731 T1831 oasc.ZkController.createCollectionZkNode
> Creating collection in ZooKeeper:collection1
>    [junit4]   2> 707732 T1831 oasc.ZkController.createCollectionZkNode Setting
> config for collection:collection1 to testWithUploadDir
>    [junit4]   2> 707732 T1831 oascc.SolrZkClient.makePath makePath:
> /collections/collection1
>    [junit4]   2> 707733 T1831 oascc.ZkStateReader.readConfigName Load
> collection config from:/collections/collection1
>    [junit4]   2> 707734 T1831 oascc.ZkStateReader.readConfigName
> path=/collections/collection1 configName=testWithUploadDir specified
> config exists in ZooKeeper
>    [junit4]   2> 707734 T1831 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-
> Solr-trunk-Linux/solr/example/solr/collection1/'
>    [junit4]   2> 707754 T1831 oasc.SolrConfig.initLibs Adding specified lib dirs to
> ClassLoader
>    [junit4]   2> 707755 T1831 oasc.SolrResourceLoader.addToClassLoader
> WARN No files added to classloader from lib: ../../../contrib/extraction/lib
> (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-
> Linux/solr/example/solr/collection1/../../../contrib/extraction/lib).
>    [junit4]   2> 707755 T1831 oasc.SolrResourceLoader.addToClassLoader
> WARN Can't find (or read) directory to add to classloader: ../../../dist/
> (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-
> Linux/solr/example/solr/collection1/../../../dist).
>    [junit4]   2> 707756 T1831 oasc.SolrResourceLoader.addToClassLoader
> WARN No files added to classloader from lib: ../../../contrib/clustering/lib/
> (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-
> Linux/solr/example/solr/collection1/../../../contrib/clustering/lib).
>    [junit4]   2> 707756 T1831 oasc.SolrResourceLoader.addToClassLoader
> WARN Can't find (or read) directory to add to classloader: ../../../dist/
> (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-
> Linux/solr/example/solr/collection1/../../../dist).
>    [junit4]   2> 707756 T1831 oasc.SolrResourceLoader.addToClassLoader
> WARN No files added to classloader from lib: ../../../contrib/langid/lib/
> (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-
> Linux/solr/example/solr/collection1/../../../contrib/langid/lib).
>    [junit4]   2> 707757 T1831 oasc.SolrResourceLoader.addToClassLoader
> WARN Can't find (or read) directory to add to classloader: ../../../dist/
> (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-
> Linux/solr/example/solr/collection1/../../../dist).
>    [junit4]   2> 707757 T1831 oasc.SolrResourceLoader.addToClassLoader
> WARN No files added to classloader from lib: ../../../contrib/velocity/lib
> (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-
> Linux/solr/example/solr/collection1/../../../contrib/velocity/lib).
>    [junit4]   2> 707757 T1831 oasc.SolrResourceLoader.addToClassLoader
> WARN Can't find (or read) directory to add to classloader: ../../../dist/
> (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-
> Linux/solr/example/solr/collection1/../../../dist).
>    [junit4]   2> 707804 T1831 oasu.SolrIndexConfig.<init> IndexWriter
> infoStream solr logging is enabled
>    [junit4]   2> 707810 T1831 oasc.SolrConfig.<init> Using Lucene
> MatchVersion: 5.0.0
>    [junit4]   2> 707880 T1831 oasc.SolrConfig.<init> Loaded SolrConfig:
> solrconfig.xml
>    [junit4]   2> 707882 T1831 oass.IndexSchema.readSchema Reading Solr
> Schema from schema.xml
>    [junit4]   2> 707888 T1831 oass.IndexSchema.readSchema [collection1]
> Schema name=example
>    [junit4]   2> 708369 T1831 oass.IndexSchema.readSchema unique key field:
> id
>    [junit4]   2> 708371 T1831
> oasr.RestManager$Registry.registerManagedResource Registered
> ManagedResource impl
> org.apache.solr.rest.schema.analysis.ManagedWordSetResource for path
> /schema/analysis/stopwords/english
>    [junit4]   2> 708371 T1831
> oasr.RestManager$Registry.registerManagedResource Registered
> ManagedResource impl
> org.apache.solr.rest.schema.analysis.ManagedSynonymFilterFactory$Synon
> ymManager for path /schema/analysis/synonyms/english
>    [junit4]   2> 708382 T1831 oass.FileExchangeRateProvider.reload Reloading
> exchange rates from file currency.xml
>    [junit4]   2> 708385 T1831 oass.FileExchangeRateProvider.reload Reloading
> exchange rates from file currency.xml
>    [junit4]   2> 708459 T1831 oasc.CoreContainer.create Creating SolrCore
> 'collection1' using configuration from collection collection1
>    [junit4]   2> 708460 T1831 oasc.SolrCore.initDirectoryFactory
> org.apache.solr.core.MockDirectoryFactory
>    [junit4]   2> 708460 T1831 oasc.SolrCore.<init> [collection1] Opening new
> SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-
> Linux/solr/example/solr/collection1/,
> dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-
> Linux/solr/example/solr/collection1/data/
>    [junit4]   2> 708460 T1831 oasc.JmxMonitoredMap.<init> JMX monitoring is
> enabled. Adding Solr mbeans to JMX Server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@2c9b730a
>    [junit4]   2> 708464 T1831 oasc.SolrCore.initListeners [collection1] Added
> SolrEventListener for newSearcher:
> org.apache.solr.core.QuerySenderListener{queries=[]}
>    [junit4]   2> 708465 T1831 oasc.SolrCore.initListeners [collection1] Added
> SolrEventListener for firstSearcher:
> org.apache.solr.core.QuerySenderListener{queries=[{q=static firstSearcher
> warming in solrconfig.xml}]}
>    [junit4]   2> 708465 T1831 oasc.SolrCore.close [collection1]  CLOSING
> SolrCore org.apache.solr.core.SolrCore@d43d142f
>    [junit4]   2> 708466 T1831 oasu.SolrCoreState.decrefSolrCoreState Closing
> SolrCoreState
>    [junit4]   2> 708466 T1831 oasu.DefaultSolrCoreState.closeIndexWriter
> SolrCoreState ref count has reached 0 - closing IndexWriter
>    [junit4]   2> 708466 T1831 oasc.SolrCore.closeSearcher [collection1] Closing
> main searcher on request.
>    [junit4]   2> 708466 T1831 oasc.CachingDirectoryFactory.close Closing
> MockDirectoryFactory - 0 directories currently being tracked
>    [junit4]   2> 708466 T1831 oasc.CoreContainer.create ERROR Error creating
> core [collection1]: Access denied ("java.io.FilePermission"
> "/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-
> Linux/solr/example/solr/collection1/data" "write")
> org.apache.solr.common.SolrException: Access denied
> ("java.io.FilePermission" "/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-
> Linux/solr/example/solr/collection1/data" "write")
>    [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:890)
>    [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
>    [junit4]   2> 	at
> org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
>    [junit4]   2> 	at
> org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
>    [junit4]   2> 	at
> org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
>    [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:273)
>    [junit4]   2> 	at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> a:1176)
>    [junit4]   2> 	at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> va:641)
>    [junit4]   2> 	at java.lang.Thread.run(Thread.java:853)
>    [junit4]   2> Caused by: java.security.AccessControlException: Access
> denied ("java.io.FilePermission" "/mnt/ssd/jenkins/workspace/Lucene-Solr-
> trunk-Linux/solr/example/solr/collection1/data" "write")
>    [junit4]   2> 	at
> java.security.AccessController.throwACE(AccessController.java:100)
>    [junit4]   2> 	at
> java.security.AccessController.checkPermission(AccessController.java:174)
>    [junit4]   2> 	at
> java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
>    [junit4]   2> 	at
> java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
>    [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
>    [junit4]   2> 	at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.j
> ava:388)
>    [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:639)
>    [junit4]   2> 	at
> java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
>    [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:692)
>    [junit4]   2> 	at
> org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
>    [junit4]   2> 	at
> org.apache.lucene.store.NativeFSLockFactory.<init>(NativeFSLockFactory.ja
> va:93)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirec
> toryFactory.java:502)
>    [junit4]   2> 	at
> org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.ja
> va:357)
>    [junit4]   2> 	at org.apache.solr.core
> 
> [...truncated too long message...]
> 
> onkeySafeLeaderTest, TestConfigSets, TestValueSourceCache,
> StressHdfsTest, ClusterStateTest, TestCollapseQParserPlugin,
> CollectionsAPIDistributedZkTest, HttpPartitionTest,
> DistributedExpandComponentTest, TestCloudPivotFacet,
> TestDynamicFieldResource, TestGroupingSearch,
> HdfsUnloadDistributedZkTest, TestComplexPhraseQParserPlugin,
> TestRestManager, CurrencyFieldXmlFileTest, TestIndexingPerformance,
> TermVectorComponentTest, TestStandardQParsers, StatsComponentTest,
> TestIBSimilarityFactory, TestMaxScoreQueryParser, TestQueryUtils,
> TestBinaryField, TestFreeTextSuggestions, TestClassNameShortening,
> TestSolrXml, ClusterStateUpdateTest, TestRandomFaceting,
> LeaderElectionTest, TestZkChroot]
>    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestZkChroot -
> Dtests.seed=E298FB1A86585611 -Dtests.multiplier=3 -Dtests.slow=true -
> Dtests.locale=sr_RS_#Latn -Dtests.timezone=America/Martinique -
> Dtests.file.encoding=ISO-8859-1
>    [junit4] ERROR   0.00s J1 | TestZkChroot (suite) <<<
>    [junit4]    > Throwable #1:
> com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in
> scope SUITE failed to close. Resource was registered from thread
> Thread[id=1831, name=coreLoadExecutor-1254-thread-1, state=RUNNABLE,
> group=TGRP-TestZkChroot], registration stack trace below.
>    [junit4]    > 	at java.lang.Thread.getStackTraceImpl(Native Method)
>    [junit4]    > 	at java.lang.Thread.getStackTrace(Thread.java:1203)
>    [junit4]    > 	at
> com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(Rando
> mizedContext.java:166)
>    [junit4]    > 	at
> org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java
> :731)
>    [junit4]    > 	at
> org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:
> 1274)
>    [junit4]    > 	at
> org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1
> 165)
>    [junit4]    > 	at
> org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1
> 157)
>    [junit4]    > 	at
> org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.jav
> a:40)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.ja
> va:351)
>    [junit4]    > 	at
> org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
>    [junit4]    > 	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
>    [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
>    [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
>    [junit4]    > 	at
> org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
>    [junit4]    > 	at
> org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
>    [junit4]    > 	at
> org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
>    [junit4]    > 	at java.util.concurrent.FutureTask.run(FutureTask.java:273)
>    [junit4]    > 	at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> a:1176)
>    [junit4]    > 	at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> va:641)
>    [junit4]    > 	at java.lang.Thread.run(Thread.java:853)
>    [junit4]    > Caused by: java.lang.AssertionError: Directory not closed:
> MockDirectoryWrapper(RAMDirectory@d8b83b0a
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6d50aeb8)
>    [junit4]    > 	at
> org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
>    [junit4]    > 	at
> com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(R
> andomizedContext.java:183)
>    [junit4]    > 	... 1 moreThrowable #2:
> com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in
> scope SUITE failed to close. Resource was registered from thread
> Thread[id=1849, name=coreLoadExecutor-1261-thread-1, state=RUNNABLE,
> group=TGRP-TestZkChroot], registration stack trace below.
>    [junit4]    > 	at java.lang.Thread.getStackTraceImpl(Native Method)
>    [junit4]    > 	at java.lang.Thread.getStackTrace(Thread.java:1203)
>    [junit4]    > 	at
> com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(Rando
> mizedContext.java:166)
>    [junit4]    > 	at
> org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java
> :731)
>    [junit4]    > 	at
> org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:
> 1274)
>    [junit4]    > 	at
> org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1
> 165)
>    [junit4]    > 	at
> org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1
> 157)
>    [junit4]    > 	at
> org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.jav
> a:40)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.ja
> va:351)
>    [junit4]    > 	at
> org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
>    [junit4]    > 	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
>    [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
>    [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
>    [junit4]    > 	at
> org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
>    [junit4]    > 	at
> org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
>    [junit4]    > 	at
> org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
>    [junit4]    > 	at java.util.concurrent.FutureTask.run(FutureTask.java:273)
>    [junit4]    > 	at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> a:1176)
>    [junit4]    > 	at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> va:641)
>    [junit4]    > 	at java.lang.Thread.run(Thread.java:853)
>    [junit4]    > Caused by: java.lang.AssertionError: Directory not closed:
> MockDirectoryWrapper(RAMDirectory@7a86ba59
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@d7401f91)
>    [junit4]    > 	at
> org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
>    [junit4]    > 	at
> com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(R
> andomizedContext.java:183)
>    [junit4]    > 	... 1 moreThrowable #3:
> com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in
> scope SUITE failed to close. Resource was registered from thread
> Thread[id=1874, name=coreLoadExecutor-1271-thread-1, state=RUNNABLE,
> group=TGRP-TestZkChroot], registration stack trace below.
>    [junit4]    > 	at java.lang.Thread.getStackTraceImpl(Native Method)
>    [junit4]    > 	at java.lang.Thread.getStackTrace(Thread.java:1203)
>    [junit4]    > 	at
> com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(Rando
> mizedContext.java:166)
>    [junit4]    > 	at
> org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java
> :731)
>    [junit4]    > 	at
> org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:
> 1274)
>    [junit4]    > 	at
> org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1
> 165)
>    [junit4]    > 	at
> org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1
> 157)
>    [junit4]    > 	at
> org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.jav
> a:40)
>    [junit4]    > 	at
> org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.ja
> va:351)
>    [junit4]    > 	at
> org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
>    [junit4]    > 	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
>    [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
>    [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
>    [junit4]    > 	at
> org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
>    [junit4]    > 	at
> org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
>    [junit4]    > 	at
> org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
>    [junit4]    > 	at java.util.concurrent.FutureTask.run(FutureTask.java:273)
>    [junit4]    > 	at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> a:1176)
>    [junit4]    > 	at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> va:641)
>    [junit4]    > 	at java.lang.Thread.run(Thread.java:853)
>    [junit4]    > Caused by: java.lang.AssertionError: Directory not closed:
> MockDirectoryWrapper(RAMDirectory@48c00581
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@de935c8b)
>    [junit4]    > 	at
> org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
>    [junit4]    > 	at
> com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(R
> andomizedContext.java:183)
>    [junit4]    > 	... 1 more
>    [junit4] Completed on J1 in 13.30s, 4 tests, 3 errors <<< FAILURES!
> 
> [...truncated 640 lines...]
> BUILD FAILED
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:491: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:484: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:60: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-
> build.xml:496: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-
> build.xml:1358: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-
> build.xml:961: There were test failures: 438 suites, 1830 tests, 3 suite-level
> errors, 2 errors, 110 ignored (23 assumptions)
> 
> Total time: 71 minutes 55 seconds
> Build step 'Invoke Ant' marked build as failure
> [description-setter] Description set: Java: 64bit/ibm-j9-jdk7 -
> Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/u
> til/fst/FST;}
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure - Any
> Sending email for trigger: Failure - Any
> 



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[JENKINS] Lucene-Solr-trunk-Linux (64bit/ibm-j9-jdk7) - Build # 11246 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11246/
Java: 64bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}

5 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestZkChroot

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=1831, name=coreLoadExecutor-1254-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=1831, name=coreLoadExecutor-1254-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
	at java.lang.Thread.getStackTraceImpl(Native Method)
	at java.lang.Thread.getStackTrace(Thread.java:1203)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:273)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1176)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
	at java.lang.Thread.run(Thread.java:853)
Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@d8b83b0a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6d50aeb8)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:699)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:696)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:712)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	... 1 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestZkChroot

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=1849, name=coreLoadExecutor-1261-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=1849, name=coreLoadExecutor-1261-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
	at java.lang.Thread.getStackTraceImpl(Native Method)
	at java.lang.Thread.getStackTrace(Thread.java:1203)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:273)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1176)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
	at java.lang.Thread.run(Thread.java:853)
Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@7a86ba59 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@d7401f91)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:699)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:696)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:712)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	... 1 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestZkChroot

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=1874, name=coreLoadExecutor-1271-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=1874, name=coreLoadExecutor-1271-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
	at java.lang.Thread.getStackTraceImpl(Native Method)
	at java.lang.Thread.getStackTrace(Thread.java:1203)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:273)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1176)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
	at java.lang.Thread.run(Thread.java:853)
Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@48c00581 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@de935c8b)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:699)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:696)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:712)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	... 1 more


FAILED:  org.apache.solr.core.CachingDirectoryFactoryTest.stressTest

Error Message:
Captured an uncaught exception in thread: Thread[id=43, name=Thread-14, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=43, name=Thread-14, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
Caused by: java.security.AccessControlException: Access denied ("java.io.FilePermission" "path17" "write")
	at __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
	at java.security.AccessController.throwACE(AccessController.java:100)
	at java.security.AccessController.checkPermission(AccessController.java:174)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
	at java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:388)
	at java.nio.file.Files.createDirectory(Files.java:639)
	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
	at java.nio.file.Files.createDirectories(Files.java:692)
	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)


FAILED:  org.apache.solr.core.RAMDirectoryFactoryTest.test

Error Message:
Access denied ("java.io.FilePermission" "/fake/path" "write")

Stack Trace:
java.security.AccessControlException: Access denied ("java.io.FilePermission" "/fake/path" "write")
	at __randomizedtesting.SeedInfo.seed([E298FB1A86585611:6ACCC4C028A43BE9]:0)
	at java.security.AccessController.throwACE(AccessController.java:100)
	at java.security.AccessController.checkPermission(AccessController.java:174)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
	at java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:388)
	at java.nio.file.Files.createDirectory(Files.java:639)
	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
	at java.nio.file.Files.createDirectories(Files.java:692)
	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
	at org.apache.solr.core.RAMDirectoryFactoryTest.dotestOpenReturnsTheSameForSamePath(RAMDirectoryFactoryTest.java:46)
	at org.apache.solr.core.RAMDirectoryFactoryTest.test(RAMDirectoryFactoryTest.java:33)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:619)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:853)




Build Log:
[...truncated 10655 lines...]
   [junit4] Suite: org.apache.solr.core.CachingDirectoryFactoryTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.CachingDirectoryFactoryTest-E298FB1A86585611-001/init-core-data-001
   [junit4]   2> 10076 T36 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 10082 T36 oas.SolrTestCaseJ4.setUp ###Starting stressTest
   [junit4]   2> 10159 T43 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@a096e6e2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@59a3485c assuming 'simple'
   [junit4]   2> 86 T43 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-14,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: Access denied ("java.io.FilePermission" "path17" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
   [junit4]   2> 	at java.security.AccessController.throwACE(AccessController.java:100)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:174)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:388)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:639)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:692)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 10163 T40 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@baae1c26 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@ae4a3824 assuming 'simple'
   [junit4]   2> 10169 T39 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@944f9e9 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4015e736 assuming 'simple'
   [junit4]   2> 93 T40 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-11,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: Access denied ("java.io.FilePermission" "path17" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
   [junit4]   2> 	at java.security.AccessController.throwACE(AccessController.java:100)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:174)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:388)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:639)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:692)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 10170 T46 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@cc8c48d1 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@46dc4ef9 assuming 'simple'
   [junit4]   2> 10172 T38 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@79cd1e49 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@d8b01404 assuming 'simple'
   [junit4]   2> 96 T46 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-17,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: Access denied ("java.io.FilePermission" "path17" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
   [junit4]   2> 	at java.security.AccessController.throwACE(AccessController.java:100)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:174)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:388)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:639)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:692)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 10173 T44 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@f3c9039c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@9747e6f assuming 'simple'
   [junit4]   2> 10175 T42 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@a59d92ee lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4ba92936 assuming 'simple'
   [junit4]   2> 94 T39 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-10,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: Access denied ("java.io.FilePermission" "path17" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
   [junit4]   2> 	at java.security.AccessController.throwACE(AccessController.java:100)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:174)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:388)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:639)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:692)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 99 T44 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-15,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: Access denied ("java.io.FilePermission" "path17" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
   [junit4]   2> 	at java.security.AccessController.throwACE(AccessController.java:100)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:174)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:388)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:639)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:692)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 10176 T37 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@3e5c575c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@9cb60008 assuming 'simple'
   [junit4]   2> 100 T42 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-13,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: Access denied ("java.io.FilePermission" "path17" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
   [junit4]   2> 	at java.security.AccessController.throwACE(AccessController.java:100)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:174)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:388)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:639)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:692)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 10179 T41 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@4b469a59 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@496d2317 assuming 'simple'
   [junit4]   2> 10182 T45 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@2d225d69 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1435a94a assuming 'simple'
   [junit4]   2> 10183 T47 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@5800c3b0 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3b869729 assuming 'simple'
   [junit4]   2> 97 T38 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-9,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: Access denied ("java.io.FilePermission" "path17" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
   [junit4]   2> 	at java.security.AccessController.throwACE(AccessController.java:100)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:174)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:388)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:639)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:692)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 107 T45 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-16,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: Access denied ("java.io.FilePermission" "path17" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
   [junit4]   2> 	at java.security.AccessController.throwACE(AccessController.java:100)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:174)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:388)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:639)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:692)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 103 T37 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-8,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: Access denied ("java.io.FilePermission" "path17" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
   [junit4]   2> 	at java.security.AccessController.throwACE(AccessController.java:100)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:174)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:388)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:639)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:692)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 106 T41 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-12,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: Access denied ("java.io.FilePermission" "path17" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
   [junit4]   2> 	at java.security.AccessController.throwACE(AccessController.java:100)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:174)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:388)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:639)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:692)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 108 T47 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-18,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: Access denied ("java.io.FilePermission" "path17" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
   [junit4]   2> 	at java.security.AccessController.throwACE(AccessController.java:100)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:174)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:388)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:639)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:692)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 18092 T54 oasc.CachingDirectoryFactory.close Closing RAMDirectoryFactory - 0 directories currently being tracked
   [junit4]   2> 18236 T36 oas.SolrTestCaseJ4.tearDown ###Ending stressTest
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CachingDirectoryFactoryTest -Dtests.method=stressTest -Dtests.seed=E298FB1A86585611 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=hu -Dtests.timezone=Asia/Thimphu -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   8.26s J1 | CachingDirectoryFactoryTest.stressTest <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=43, name=Thread-14, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: Access denied ("java.io.FilePermission" "path17" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
   [junit4]    > 	at java.security.AccessController.throwACE(AccessController.java:100)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:174)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:388)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:639)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:692)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=40, name=Thread-11, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: Access denied ("java.io.FilePermission" "path17" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
   [junit4]    > 	at java.security.AccessController.throwACE(AccessController.java:100)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:174)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:388)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:639)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:692)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=39, name=Thread-10, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: Access denied ("java.io.FilePermission" "path17" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
   [junit4]    > 	at java.security.AccessController.throwACE(AccessController.java:100)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:174)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:388)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:639)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:692)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #4: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=46, name=Thread-17, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: Access denied ("java.io.FilePermission" "path17" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
   [junit4]    > 	at java.security.AccessController.throwACE(AccessController.java:100)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:174)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:388)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:639)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:692)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #5: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=38, name=Thread-9, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: Access denied ("java.io.FilePermission" "path17" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
   [junit4]    > 	at java.security.AccessController.throwACE(AccessController.java:100)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:174)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:388)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:639)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:692)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #6: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=44, name=Thread-15, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: Access denied ("java.io.FilePermission" "path17" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
   [junit4]    > 	at java.security.AccessController.throwACE(AccessController.java:100)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:174)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:388)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:639)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:692)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #7: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=42, name=Thread-13, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: Access denied ("java.io.FilePermission" "path17" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
   [junit4]    > 	at java.security.AccessController.throwACE(AccessController.java:100)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:174)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:388)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:639)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:692)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #8: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=37, name=Thread-8, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: Access denied ("java.io.FilePermission" "path17" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
   [junit4]    > 	at java.security.AccessController.throwACE(AccessController.java:100)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:174)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:388)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:639)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:692)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #9: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=41, name=Thread-12, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: Access denied ("java.io.FilePermission" "path17" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
   [junit4]    > 	at java.security.AccessController.throwACE(AccessController.java:100)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:174)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:388)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:639)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:692)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #10: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=45, name=Thread-16, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: Access denied ("java.io.FilePermission" "path17" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
   [junit4]    > 	at java.security.AccessController.throwACE(AccessController.java:100)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:174)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:388)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:639)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:692)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #11: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=47, name=Thread-18, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: Access denied ("java.io.FilePermission" "path17" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E298FB1A86585611]:0)
   [junit4]    > 	at java.security.AccessController.throwACE(AccessController.java:100)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:174)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:388)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:639)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:692)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 18333 T36 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=318), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=318)), sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=hu, timezone=Asia/Thimphu
   [junit4]   2> NOTE: Linux 3.13.0-35-generic amd64/IBM Corporation 1.7.0 (64-bit)/cpus=8,threads=1,free=7770776,total=35061760
   [junit4]   2> NOTE: All tests run in this JVM: [OutputWriterTest, SpatialFilterTest, CachingDirectoryFactoryTest]
   [junit4] Completed on J1 in 8.38s, 1 test, 1 error <<< FAILURES!

[...truncated 113 lines...]
   [junit4] Suite: org.apache.solr.core.RAMDirectoryFactoryTest
   [junit4]   2> 95703 T322 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@ff26adfe lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@f6628f14 assuming 'simple'
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=RAMDirectoryFactoryTest -Dtests.method=test -Dtests.seed=E298FB1A86585611 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=sh_CS -Dtests.timezone=Africa/Lagos -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.01s J1 | RAMDirectoryFactoryTest.test <<<
   [junit4]    > Throwable #1: java.security.AccessControlException: Access denied ("java.io.FilePermission" "/fake/path" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E298FB1A86585611:6ACCC4C028A43BE9]:0)
   [junit4]    > 	at java.security.AccessController.throwACE(AccessController.java:100)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:174)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:388)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:639)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:692)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.RAMDirectoryFactoryTest.dotestOpenReturnsTheSameForSamePath(RAMDirectoryFactoryTest.java:46)
   [junit4]    > 	at org.apache.solr.core.RAMDirectoryFactoryTest.test(RAMDirectoryFactoryTest.java:33)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:853)
   [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{}, sim=DefaultSimilarity, locale=sh_CS, timezone=Africa/Lagos
   [junit4]   2> NOTE: Linux 3.13.0-35-generic amd64/IBM Corporation 1.7.0 (64-bit)/cpus=8,threads=1,free=24885040,total=85917696
   [junit4]   2> NOTE: All tests run in this JVM: [OutputWriterTest, SpatialFilterTest, CachingDirectoryFactoryTest, HdfsDirectoryTest, TestCopyFieldCollectionResource, OpenCloseCoreStressTest, NumericFieldsTest, QueryElevationComponentTest, DistributedQueryElevationComponentTest, CacheHeaderTest, BasicDistributedZk2Test, TestReplicationHandlerBackup, SliceStateTest, AutoCommitTest, OverriddenZkACLAndCredentialsProvidersTest, CoreMergeIndexesAdminHandlerTest, UpdateRequestProcessorFactoryTest, LegacyHTMLStripCharFilterTest, ScriptEngineTest, RAMDirectoryFactoryTest]
   [junit4] Completed on J1 in 0.03s, 1 test, 1 error <<< FAILURES!

[...truncated 579 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestZkChroot
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.TestZkChroot-E298FB1A86585611-001/init-core-data-001
   [junit4]   2> 706204 T1816 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 706207 T1816 oas.SolrTestCaseJ4.setUp ###Starting testWithUploadDir
   [junit4]   2> 706207 T1816 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 706208 T1817 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 706308 T1816 oasc.ZkTestServer.run start zk server on port:44091
   [junit4]   2> 706309 T1816 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 706310 T1816 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 706313 T1823 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4e404da3 name:ZooKeeperConnection Watcher:127.0.0.1:44091 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 706313 T1816 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 706314 T1816 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 706315 T1816 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/'
   [junit4]   2> 706328 T1816 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/solr.xml
   [junit4]   2> 706359 T1816 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr
   [junit4]   2> 706360 T1816 oasc.CoreContainer.<init> New CoreContainer -225327910
   [junit4]   2> 706360 T1816 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/]
   [junit4]   2> 706361 T1816 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 706362 T1816 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 706362 T1816 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 706362 T1816 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 706363 T1816 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 706363 T1816 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 706363 T1816 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 706364 T1816 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 706364 T1816 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 706366 T1816 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=0&connTimeout=0&retry=false
   [junit4]   2> 706367 T1816 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 706367 T1816 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 706367 T1816 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 706368 T1816 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:44091/foo/bar3
   [junit4]   2> 706368 T1816 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 706368 T1816 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 706369 T1816 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 706370 T1825 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6ff14d4d name:ZooKeeperConnection Watcher:127.0.0.1:44091 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 706371 T1816 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 706371 T1816 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 706372 T1816 oascc.SolrZkClient.makePath makePath: /foo/bar3
   [junit4]   2> 706377 T1816 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 706378 T1827 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@bbbbf8aa name:ZooKeeperConnection Watcher:127.0.0.1:44091/foo/bar3 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 706378 T1816 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 706379 T1816 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 706383 T1816 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 706386 T1816 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 706389 T1816 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 706392 T1816 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 706395 T1816 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 706397 T1816 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/188.138.97.18:8983_solr
   [junit4]   2> 706398 T1816 oascc.SolrZkClient.makePath makePath: /live_nodes/188.138.97.18:8983_solr
   [junit4]   2> 706400 T1816 oascc.SolrZkClient.makePath makePath: /collections
   [junit4]   2> 706402 T1816 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 706404 T1816 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 706406 T1816 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 706408 T1816 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 188.138.97.18:8983_solr
   [junit4]   2> 706408 T1816 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 706410 T1816 oasc.Overseer.start Overseer (id=92453081980010498-188.138.97.18:8983_solr-n_0000000000) starting
   [junit4]   2> 706413 T1816 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 706423 T1816 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 706425 T1829 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 706425 T1816 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 706429 T1816 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 706430 T1816 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 706433 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/spellings.txt
   [junit4]   2> 706433 T1828 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 706435 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/stopwords.txt
   [junit4]   2> 706438 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/scripts.conf
   [junit4]   2> 706439 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/admin-extra.html
   [junit4]   2> 706441 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/schema.xml
   [junit4]   2> 706443 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/solrconfig.xml
   [junit4]   2> 706446 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/admin-extra.menu-top.html
   [junit4]   2> 706448 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/elevate.xml
   [junit4]   2> 706451 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/_schema_analysis_synonyms_english.json
   [junit4]   2> 706454 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_nl.txt
   [junit4]   2> 706457 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_eu.txt
   [junit4]   2> 706459 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_ga.txt
   [junit4]   2> 706462 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_fr.txt
   [junit4]   2> 706465 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_bg.txt
   [junit4]   2> 706468 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stemdict_nl.txt
   [junit4]   2> 706471 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/contractions_ca.txt
   [junit4]   2> 706474 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_ja.txt
   [junit4]   2> 706477 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_id.txt
   [junit4]   2> 706481 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_it.txt
   [junit4]   2> 706485 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_fi.txt
   [junit4]   2> 706488 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/contractions_fr.txt
   [junit4]   2> 706491 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_el.txt
   [junit4]   2> 706494 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_en.txt
   [junit4]   2> 706497 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_da.txt
   [junit4]   2> 706500 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stoptags_ja.txt
   [junit4]   2> 706503 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_no.txt
   [junit4]   2> 706505 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_pt.txt
   [junit4]   2> 706509 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/hyphenations_ga.txt
   [junit4]   2> 706513 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_ru.txt
   [junit4]   2> 706517 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_ro.txt
   [junit4]   2> 706520 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_cz.txt
   [junit4]   2> 706524 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_hu.txt
   [junit4]   2> 706528 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_th.txt
   [junit4]   2> 706531 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_ckb.txt
   [junit4]   2> 706535 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_gl.txt
   [junit4]   2> 706538 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_ar.txt
   [junit4]   2> 706542 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_tr.txt
   [junit4]   2> 706545 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_es.txt
   [junit4]   2> 706549 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/userdict_ja.txt
   [junit4]   2> 706552 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/contractions_it.txt
   [junit4]   2> 706556 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_ca.txt
   [junit4]   2> 706559 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_lv.txt
   [junit4]   2> 706563 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_de.txt
   [junit4]   2> 706566 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/contractions_ga.txt
   [junit4]   2> 706569 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_hi.txt
   [junit4]   2> 706572 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_hy.txt
   [junit4]   2> 706574 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_sv.txt
   [junit4]   2> 706577 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_fa.txt
   [junit4]   2> 706579 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/synonyms.txt
   [junit4]   2> 706582 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/update-script.js
   [junit4]   2> 706584 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/facet_pivot.vm
   [junit4]   2> 706587 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/jquery.autocomplete.css
   [junit4]   2> 706591 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/debug.vm
   [junit4]   2> 706594 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/query_form.vm
   [junit4]   2> 706597 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/pagination_top.vm
   [junit4]   2> 706601 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/did_you_mean.vm
   [junit4]   2> 706604 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/query_spatial.vm
   [junit4]   2> 706608 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/hit_plain.vm
   [junit4]   2> 706611 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/pagination_bottom.vm
   [junit4]   2> 706614 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/header.vm
   [junit4]   2> 706618 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/layout.vm
   [junit4]   2> 706622 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/suggest.vm
   [junit4]   2> 706625 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/README.txt
   [junit4]   2> 706629 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/head.vm
   [junit4]   2> 706633 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/footer.vm
   [junit4]   2> 706637 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/results_list.vm
   [junit4]   2> 706640 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/jquery.autocomplete.js
   [junit4]   2> 706644 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/hit_grouped.vm
   [junit4]   2> 706647 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/facet_fields.vm
   [junit4]   2> 706649 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/query_group.vm
   [junit4]   2> 706650 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/query.vm
   [junit4]   2> 706652 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/cluster_results.vm
   [junit4]   2> 706654 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/VM_global_library.vm
   [junit4]   2> 706656 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/facet_queries.vm
   [junit4]   2> 706657 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/mime_type_lists.vm
   [junit4]   2> 706659 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/error.vm
   [junit4]   2> 706661 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/product_doc.vm
   [junit4]   2> 706662 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/main.css
   [junit4]   2> 706664 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/browse.vm
   [junit4]   2> 706666 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/richtext_doc.vm
   [junit4]   2> 706669 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/cluster.vm
   [junit4]   2> 706673 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/facets.vm
   [junit4]   2> 706677 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/tabs.vm
   [junit4]   2> 706681 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/join_doc.vm
   [junit4]   2> 706684 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/hit.vm
   [junit4]   2> 706687 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/facet_ranges.vm
   [junit4]   2> 706691 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/mapping-ISOLatin1Accent.txt
   [junit4]   2> 706693 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/_schema_analysis_stopwords_english.json
   [junit4]   2> 706696 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/protwords.txt
   [junit4]   2> 706699 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/_rest_managed.json
   [junit4]   2> 706701 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/xslt/example.xsl
   [junit4]   2> 706703 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/xslt/example_rss.xsl
   [junit4]   2> 706704 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/xslt/updateXml.xsl
   [junit4]   2> 706706 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/xslt/luke.xsl
   [junit4]   2> 706707 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/xslt/example_atom.xsl
   [junit4]   2> 706709 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/mapping-FoldToASCII.txt
   [junit4]   2> 706710 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/clustering/carrot2/stc-attributes.xml
   [junit4]   2> 706713 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/clustering/carrot2/lingo-attributes.xml
   [junit4]   2> 706716 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/clustering/carrot2/kmeans-attributes.xml
   [junit4]   2> 706718 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/admin-extra.menu-bottom.html
   [junit4]   2> 706720 T1816 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/currency.xml
   [junit4]   2> 706724 T1816 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr
   [junit4]   2> 706727 T1816 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/
   [junit4]   2> 706727 T1816 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 706728 T1831 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 706728 T1831 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 706730 T1831 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 706730 T1832 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 706733 T1828 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "base_url":"http://188.138.97.18:8983/solr",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "node_name":"188.138.97.18:8983_solr",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "state":"down"}
   [junit4]   2> 706734 T1828 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 706738 T1832 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 707730 T1831 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 707731 T1831 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 707731 T1831 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:collection1
   [junit4]   2> 707732 T1831 oasc.ZkController.createCollectionZkNode Setting config for collection:collection1 to testWithUploadDir
   [junit4]   2> 707732 T1831 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 707733 T1831 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 707734 T1831 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=testWithUploadDir specified config exists in ZooKeeper
   [junit4]   2> 707734 T1831 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/'
   [junit4]   2> 707754 T1831 oasc.SolrConfig.initLibs Adding specified lib dirs to ClassLoader
   [junit4]   2> 707755 T1831 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/extraction/lib (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/extraction/lib).
   [junit4]   2> 707755 T1831 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 707756 T1831 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/clustering/lib/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/clustering/lib).
   [junit4]   2> 707756 T1831 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 707756 T1831 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/langid/lib/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/langid/lib).
   [junit4]   2> 707757 T1831 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 707757 T1831 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/velocity/lib (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/velocity/lib).
   [junit4]   2> 707757 T1831 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 707804 T1831 oasu.SolrIndexConfig.<init> IndexWriter infoStream solr logging is enabled
   [junit4]   2> 707810 T1831 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 707880 T1831 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 707882 T1831 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 707888 T1831 oass.IndexSchema.readSchema [collection1] Schema name=example
   [junit4]   2> 708369 T1831 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 708371 T1831 oasr.RestManager$Registry.registerManagedResource Registered ManagedResource impl org.apache.solr.rest.schema.analysis.ManagedWordSetResource for path /schema/analysis/stopwords/english
   [junit4]   2> 708371 T1831 oasr.RestManager$Registry.registerManagedResource Registered ManagedResource impl org.apache.solr.rest.schema.analysis.ManagedSynonymFilterFactory$SynonymManager for path /schema/analysis/synonyms/english
   [junit4]   2> 708382 T1831 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 708385 T1831 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 708459 T1831 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 708460 T1831 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 708460 T1831 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data/
   [junit4]   2> 708460 T1831 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c9b730a
   [junit4]   2> 708464 T1831 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for newSearcher: org.apache.solr.core.QuerySenderListener{queries=[]}
   [junit4]   2> 708465 T1831 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for firstSearcher: org.apache.solr.core.QuerySenderListener{queries=[{q=static firstSearcher warming in solrconfig.xml}]}
   [junit4]   2> 708465 T1831 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@d43d142f
   [junit4]   2> 708466 T1831 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 708466 T1831 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 708466 T1831 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 708466 T1831 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 0 directories currently being tracked
   [junit4]   2> 708466 T1831 oasc.CoreContainer.create ERROR Error creating core [collection1]: Access denied ("java.io.FilePermission" "/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data" "write") org.apache.solr.common.SolrException: Access denied ("java.io.FilePermission" "/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data" "write")
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:890)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:273)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1176)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:853)
   [junit4]   2> Caused by: java.security.AccessControlException: Access denied ("java.io.FilePermission" "/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data" "write")
   [junit4]   2> 	at java.security.AccessController.throwACE(AccessController.java:100)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:174)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:992)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:816)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:388)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:639)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:743)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:692)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.NativeFSLockFactory.<init>(NativeFSLockFactory.java:93)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:502)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core

[...truncated too long message...]

onkeySafeLeaderTest, TestConfigSets, TestValueSourceCache, StressHdfsTest, ClusterStateTest, TestCollapseQParserPlugin, CollectionsAPIDistributedZkTest, HttpPartitionTest, DistributedExpandComponentTest, TestCloudPivotFacet, TestDynamicFieldResource, TestGroupingSearch, HdfsUnloadDistributedZkTest, TestComplexPhraseQParserPlugin, TestRestManager, CurrencyFieldXmlFileTest, TestIndexingPerformance, TermVectorComponentTest, TestStandardQParsers, StatsComponentTest, TestIBSimilarityFactory, TestMaxScoreQueryParser, TestQueryUtils, TestBinaryField, TestFreeTextSuggestions, TestClassNameShortening, TestSolrXml, ClusterStateUpdateTest, TestRandomFaceting, LeaderElectionTest, TestZkChroot]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestZkChroot -Dtests.seed=E298FB1A86585611 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=sr_RS_#Latn -Dtests.timezone=America/Martinique -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J1 | TestZkChroot (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=1831, name=coreLoadExecutor-1254-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
   [junit4]    > 	at java.lang.Thread.getStackTraceImpl(Native Method)
   [junit4]    > 	at java.lang.Thread.getStackTrace(Thread.java:1203)
   [junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
   [junit4]    > 	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
   [junit4]    > 	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
   [junit4]    > 	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
   [junit4]    > 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
   [junit4]    > 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
   [junit4]    > 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
   [junit4]    > 	at java.util.concurrent.FutureTask.run(FutureTask.java:273)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1176)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:853)
   [junit4]    > Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@d8b83b0a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6d50aeb8)
   [junit4]    > 	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
   [junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
   [junit4]    > 	... 1 moreThrowable #2: com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=1849, name=coreLoadExecutor-1261-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
   [junit4]    > 	at java.lang.Thread.getStackTraceImpl(Native Method)
   [junit4]    > 	at java.lang.Thread.getStackTrace(Thread.java:1203)
   [junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
   [junit4]    > 	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
   [junit4]    > 	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
   [junit4]    > 	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
   [junit4]    > 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
   [junit4]    > 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
   [junit4]    > 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
   [junit4]    > 	at java.util.concurrent.FutureTask.run(FutureTask.java:273)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1176)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:853)
   [junit4]    > Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@7a86ba59 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@d7401f91)
   [junit4]    > 	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
   [junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
   [junit4]    > 	... 1 moreThrowable #3: com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=1874, name=coreLoadExecutor-1271-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
   [junit4]    > 	at java.lang.Thread.getStackTraceImpl(Native Method)
   [junit4]    > 	at java.lang.Thread.getStackTrace(Thread.java:1203)
   [junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
   [junit4]    > 	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
   [junit4]    > 	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
   [junit4]    > 	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
   [junit4]    > 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
   [junit4]    > 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
   [junit4]    > 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
   [junit4]    > 	at java.util.concurrent.FutureTask.run(FutureTask.java:273)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1176)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:853)
   [junit4]    > Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@48c00581 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@de935c8b)
   [junit4]    > 	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
   [junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
   [junit4]    > 	... 1 more
   [junit4] Completed on J1 in 13.30s, 4 tests, 3 errors <<< FAILURES!

[...truncated 640 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:484: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:60: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1358: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:961: There were test failures: 438 suites, 1830 tests, 3 suite-level errors, 2 errors, 110 ignored (23 assumptions)

Total time: 71 minutes 55 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_20) - Build # 11245 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11245/
Java: 64bit/jdk1.8.0_20 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

5 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestZkChroot

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=1980, name=coreLoadExecutor-1052-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=1980, name=coreLoadExecutor-1052-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
	at java.lang.Thread.getStackTrace(Thread.java:1552)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@467ac695 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5e0d295f)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:699)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:696)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:712)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	... 1 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestZkChroot

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=2000, name=coreLoadExecutor-1059-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=2000, name=coreLoadExecutor-1059-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
	at java.lang.Thread.getStackTrace(Thread.java:1552)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@4d02581b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7b19cce0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:699)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:696)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:712)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	... 1 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestZkChroot

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=2018, name=coreLoadExecutor-1066-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=2018, name=coreLoadExecutor-1066-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
	at java.lang.Thread.getStackTrace(Thread.java:1552)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@5fa8f71c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4ce9550d)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:699)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:696)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:712)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	... 1 more


FAILED:  org.apache.solr.core.CachingDirectoryFactoryTest.stressTest

Error Message:
Captured an uncaught exception in thread: Thread[id=7466, name=Thread-2785, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=7466, name=Thread-2785, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path18/8/1" "write")
	at __randomizedtesting.SeedInfo.seed([3D70C21FACD02AF3]:0)
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
	at java.security.AccessController.checkPermission(AccessController.java:884)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
	at java.nio.file.Files.createDirectory(Files.java:674)
	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
	at java.nio.file.Files.createDirectories(Files.java:727)
	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)


FAILED:  org.apache.solr.core.RAMDirectoryFactoryTest.test

Error Message:
access denied ("java.io.FilePermission" "/fake/path" "write")

Stack Trace:
java.security.AccessControlException: access denied ("java.io.FilePermission" "/fake/path" "write")
	at __randomizedtesting.SeedInfo.seed([3D70C21FACD02AF3:B524FDC5022C470B]:0)
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
	at java.security.AccessController.checkPermission(AccessController.java:884)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
	at java.nio.file.Files.createDirectory(Files.java:674)
	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
	at java.nio.file.Files.createDirectories(Files.java:727)
	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
	at org.apache.solr.core.RAMDirectoryFactoryTest.dotestOpenReturnsTheSameForSamePath(RAMDirectoryFactoryTest.java:46)
	at org.apache.solr.core.RAMDirectoryFactoryTest.test(RAMDirectoryFactoryTest.java:33)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 10866 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestZkChroot
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.TestZkChroot-3D70C21FACD02AF3-001/init-core-data-001
   [junit4]   2> 565127 T1958 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 565131 T1958 oas.SolrTestCaseJ4.setUp ###Starting testNoBootstrapConf
   [junit4]   2> 565131 T1958 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 565132 T1959 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 565232 T1958 oasc.ZkTestServer.run start zk server on port:40641
   [junit4]   2> 565233 T1958 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 565234 T1958 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 565238 T1965 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c87d2e7 name:ZooKeeperConnection Watcher:127.0.0.1:40641 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 565238 T1958 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 565239 T1958 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 565240 T1958 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/'
   [junit4]   2> 565256 T1958 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/solr.xml
   [junit4]   2> 565263 T1958 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr
   [junit4]   2> 565264 T1958 oasc.CoreContainer.<init> New CoreContainer 1966091401
   [junit4]   2> 565264 T1958 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/]
   [junit4]   2> 565265 T1958 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 565265 T1958 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 565265 T1958 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 565266 T1958 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 565266 T1958 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 565266 T1958 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 565267 T1958 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 565267 T1958 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 565267 T1958 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 565269 T1958 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=0&connTimeout=0&retry=false
   [junit4]   2> 565270 T1958 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 565271 T1958 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 565271 T1958 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 565271 T1958 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40641/foo/bar2
   [junit4]   2> 565272 T1958 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 565272 T1958 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 565273 T1958 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 565274 T1967 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@736383d8 name:ZooKeeperConnection Watcher:127.0.0.1:40641 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 565276 T1958 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 565276 T1958 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 565279 T1960 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14874dce38f0001, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 565282 T1958 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:40641 40641
   [junit4]   2> 565845 T1959 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:40641 40641
   [junit4]   2> 565848 T1958 oas.SolrTestCaseJ4.tearDown ###Ending testNoBootstrapConf
   [junit4]   2> 565857 T1958 oas.SolrTestCaseJ4.setUp ###Starting testChrootBootstrap
   [junit4]   2> 565858 T1958 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 565859 T1968 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 565959 T1958 oasc.ZkTestServer.run start zk server on port:36560
   [junit4]   2> 565960 T1958 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/'
   [junit4]   2> 565975 T1958 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/solr.xml
   [junit4]   2> 565985 T3 oasc.CoreContainer.finalize ERROR CoreContainer was not close prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!  instance=1966091401
   [junit4]   2> 565987 T1958 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr
   [junit4]   2> 565987 T1958 oasc.CoreContainer.<init> New CoreContainer 1342252620
   [junit4]   2> 565987 T1958 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/]
   [junit4]   2> 565988 T1958 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 565988 T1958 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 565988 T1958 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 565988 T1958 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 565988 T1958 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 565989 T1958 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 565989 T1958 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 565989 T1958 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 565989 T1958 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 565990 T1958 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=0&connTimeout=0&retry=false
   [junit4]   2> 565991 T1958 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 565991 T1958 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 565991 T1958 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 565991 T1958 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:36560/foo/bar
   [junit4]   2> 565992 T1958 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 565992 T1958 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 565992 T1958 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 565995 T1974 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7d1ae1b name:ZooKeeperConnection Watcher:127.0.0.1:36560 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 565996 T1958 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 565997 T1958 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 565998 T1958 oascc.SolrZkClient.makePath makePath: /foo/bar
   [junit4]   2> 566005 T1958 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 566006 T1976 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3e77722f name:ZooKeeperConnection Watcher:127.0.0.1:36560/foo/bar got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 566006 T1958 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 566007 T1958 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 566010 T1958 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 566011 T1958 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 566013 T1958 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 566014 T1958 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 566015 T1958 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 566015 T1958 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/188.138.97.18:8983_solr
   [junit4]   2> 566016 T1958 oascc.SolrZkClient.makePath makePath: /live_nodes/188.138.97.18:8983_solr
   [junit4]   2> 566017 T1958 oascc.SolrZkClient.makePath makePath: /collections
   [junit4]   2> 566018 T1958 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 566019 T1958 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 566020 T1958 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 566021 T1958 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 188.138.97.18:8983_solr
   [junit4]   2> 566021 T1958 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 566022 T1958 oasc.Overseer.start Overseer (id=92452284468166657-188.138.97.18:8983_solr-n_0000000000) starting
   [junit4]   2> 566023 T1958 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 566026 T1958 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 566027 T1978 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 566027 T1958 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 566029 T1958 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 566030 T1958 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 566033 T1958 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr
   [junit4]   2> 566033 T1977 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 566037 T1958 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/
   [junit4]   2> 566038 T1958 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 566038 T1958 oasc.ZkController.bootstrapConf bootstrapping config for 1 cores into ZooKeeper using solr.xml from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/
   [junit4]   2> 566039 T1958 oasc.ZkController.bootstrapConf Uploading directory /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/conf with name collection1 for SolrCore collection1
   [junit4]   2> 566042 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/spellings.txt
   [junit4]   2> 566046 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/stopwords.txt
   [junit4]   2> 566050 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/scripts.conf
   [junit4]   2> 566053 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/admin-extra.html
   [junit4]   2> 566057 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/schema.xml
   [junit4]   2> 566061 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/solrconfig.xml
   [junit4]   2> 566065 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/admin-extra.menu-top.html
   [junit4]   2> 566069 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/elevate.xml
   [junit4]   2> 566073 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/_schema_analysis_synonyms_english.json
   [junit4]   2> 566077 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_nl.txt
   [junit4]   2> 566082 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_eu.txt
   [junit4]   2> 566085 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ga.txt
   [junit4]   2> 566089 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_fr.txt
   [junit4]   2> 566093 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_bg.txt
   [junit4]   2> 566098 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stemdict_nl.txt
   [junit4]   2> 566102 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/contractions_ca.txt
   [junit4]   2> 566105 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ja.txt
   [junit4]   2> 566108 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_id.txt
   [junit4]   2> 566111 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_it.txt
   [junit4]   2> 566114 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_fi.txt
   [junit4]   2> 566118 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/contractions_fr.txt
   [junit4]   2> 566121 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_el.txt
   [junit4]   2> 566123 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_en.txt
   [junit4]   2> 566125 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_da.txt
   [junit4]   2> 566126 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stoptags_ja.txt
   [junit4]   2> 566128 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_no.txt
   [junit4]   2> 566129 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_pt.txt
   [junit4]   2> 566131 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/hyphenations_ga.txt
   [junit4]   2> 566132 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ru.txt
   [junit4]   2> 566134 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ro.txt
   [junit4]   2> 566135 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_cz.txt
   [junit4]   2> 566137 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_hu.txt
   [junit4]   2> 566139 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_th.txt
   [junit4]   2> 566140 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ckb.txt
   [junit4]   2> 566142 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_gl.txt
   [junit4]   2> 566145 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ar.txt
   [junit4]   2> 566147 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_tr.txt
   [junit4]   2> 566150 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_es.txt
   [junit4]   2> 566152 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/userdict_ja.txt
   [junit4]   2> 566155 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/contractions_it.txt
   [junit4]   2> 566157 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ca.txt
   [junit4]   2> 566160 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_lv.txt
   [junit4]   2> 566163 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_de.txt
   [junit4]   2> 566166 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/contractions_ga.txt
   [junit4]   2> 566169 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_hi.txt
   [junit4]   2> 566171 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_hy.txt
   [junit4]   2> 566174 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_sv.txt
   [junit4]   2> 566176 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_fa.txt
   [junit4]   2> 566178 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/synonyms.txt
   [junit4]   2> 566180 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/update-script.js
   [junit4]   2> 566182 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facet_pivot.vm
   [junit4]   2> 566185 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/jquery.autocomplete.css
   [junit4]   2> 566188 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/debug.vm
   [junit4]   2> 566191 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/query_form.vm
   [junit4]   2> 566194 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/pagination_top.vm
   [junit4]   2> 566196 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/did_you_mean.vm
   [junit4]   2> 566199 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/query_spatial.vm
   [junit4]   2> 566202 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/hit_plain.vm
   [junit4]   2> 566205 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/pagination_bottom.vm
   [junit4]   2> 566207 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/header.vm
   [junit4]   2> 566210 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/layout.vm
   [junit4]   2> 566212 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/suggest.vm
   [junit4]   2> 566215 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/README.txt
   [junit4]   2> 566218 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/head.vm
   [junit4]   2> 566220 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/footer.vm
   [junit4]   2> 566223 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/results_list.vm
   [junit4]   2> 566226 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/jquery.autocomplete.js
   [junit4]   2> 566229 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/hit_grouped.vm
   [junit4]   2> 566233 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facet_fields.vm
   [junit4]   2> 566237 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/query_group.vm
   [junit4]   2> 566241 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/query.vm
   [junit4]   2> 566244 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/cluster_results.vm
   [junit4]   2> 566248 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/VM_global_library.vm
   [junit4]   2> 566250 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facet_queries.vm
   [junit4]   2> 566253 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/mime_type_lists.vm
   [junit4]   2> 566255 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/error.vm
   [junit4]   2> 566257 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/product_doc.vm
   [junit4]   2> 566260 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/main.css
   [junit4]   2> 566262 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/browse.vm
   [junit4]   2> 566264 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/richtext_doc.vm
   [junit4]   2> 566266 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/cluster.vm
   [junit4]   2> 566269 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facets.vm
   [junit4]   2> 566272 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/tabs.vm
   [junit4]   2> 566274 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/join_doc.vm
   [junit4]   2> 566277 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/hit.vm
   [junit4]   2> 566281 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facet_ranges.vm
   [junit4]   2> 566284 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 566286 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/_schema_analysis_stopwords_english.json
   [junit4]   2> 566289 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/protwords.txt
   [junit4]   2> 566292 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/_rest_managed.json
   [junit4]   2> 566295 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/example.xsl
   [junit4]   2> 566298 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/example_rss.xsl
   [junit4]   2> 566301 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/updateXml.xsl
   [junit4]   2> 566304 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/luke.xsl
   [junit4]   2> 566305 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/example_atom.xsl
   [junit4]   2> 566307 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/mapping-FoldToASCII.txt
   [junit4]   2> 566311 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/clustering/carrot2/stc-attributes.xml
   [junit4]   2> 566314 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/clustering/carrot2/lingo-attributes.xml
   [junit4]   2> 566317 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/clustering/carrot2/kmeans-attributes.xml
   [junit4]   2> 566319 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/admin-extra.menu-bottom.html
   [junit4]   2> 566321 T1958 oascc.SolrZkClient.makePath makePath: /configs/collection1/currency.xml
   [junit4]   2> 566324 T1958 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr
   [junit4]   2> 566326 T1958 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/
   [junit4]   2> 566327 T1958 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 566328 T1980 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 566328 T1980 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 566329 T1980 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 566329 T1981 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 566332 T1977 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://188.138.97.18:8983/solr",
   [junit4]   2> 	  "node_name":"188.138.97.18:8983_solr",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 566332 T1977 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 566335 T1981 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 567329 T1980 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 567330 T1980 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 567330 T1980 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:collection1
   [junit4]   2> 567331 T1980 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 567333 T1980 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 567334 T1980 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=collection1 specified config exists in ZooKeeper
   [junit4]   2> 567334 T1980 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/'
   [junit4]   2> 567357 T1980 oasc.SolrConfig.initLibs Adding specified lib dirs to ClassLoader
   [junit4]   2> 567358 T1980 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/extraction/lib (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/extraction/lib).
   [junit4]   2> 567358 T1980 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 567359 T1980 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/clustering/lib/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/clustering/lib).
   [junit4]   2> 567359 T1980 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 567359 T1980 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/langid/lib/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/langid/lib).
   [junit4]   2> 567359 T1980 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 567359 T1980 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/velocity/lib (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/velocity/lib).
   [junit4]   2> 567360 T1980 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 567382 T1980 oasu.SolrIndexConfig.<init> IndexWriter infoStream solr logging is enabled
   [junit4]   2> 567384 T1980 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 567421 T1980 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 567426 T1980 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 567429 T1980 oass.IndexSchema.readSchema [collection1] Schema name=example
   [junit4]   2> 567658 T1980 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 567660 T1980 oasr.RestManager$Registry.registerManagedResource Registered ManagedResource impl org.apache.solr.rest.schema.analysis.ManagedWordSetResource for path /schema/analysis/stopwords/english
   [junit4]   2> 567660 T1980 oasr.RestManager$Registry.registerManagedResource Registered ManagedResource impl org.apache.solr.rest.schema.analysis.ManagedSynonymFilterFactory$SynonymManager for path /schema/analysis/synonyms/english
   [junit4]   2> 567670 T1980 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 567674 T1980 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 567788 T1980 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 567789 T1980 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 567789 T1980 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data/
   [junit4]   2> 567790 T1980 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7de237b8
   [junit4]   2> 567801 T1980 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for newSearcher: org.apache.solr.core.QuerySenderListener{queries=[]}
   [junit4]   2> 567801 T1980 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for firstSearcher: org.apache.solr.core.QuerySenderListener{queries=[{q=static firstSearcher warming in solrconfig.xml}]}
   [junit4]   2> 567802 T1980 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@3c212cd1
   [junit4]   2> 567802 T1980 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 567803 T1980 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 567803 T1980 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 567803 T1980 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 0 directories currently being tracked
   [junit4]   2> 567804 T1980 oasc.CoreContainer.create ERROR Error creating core [collection1]: access denied ("java.io.FilePermission" "/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data" "write") org.apache.solr.common.SolrException: access denied ("java.io.FilePermission" "/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data" "write")
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:891)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data" "write")
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.NativeFSLockFactory.<init>(NativeFSLockFactory.java:93)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:502)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
   [junit4]   2> 	... 8 more
   [junit4]   2> 
   [junit4]   2> 567806 T1958 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 567807 T1958 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 567809 T1984 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@34282452 name:ZooKeeperConnection Watcher:127.0.0.1:36560 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 567810 T1958 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 567810 T1958 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 567811 T1958 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1342252620
   [junit4]   2> 567811 T1958 oasc.Overseer.close Overseer (id=92452284468166657-188.138.97.18:8983_solr-n_0000000000) closing
   [junit4]   2> 567812 T1977 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 188.138.97.18:8983_solr
   [junit4]   2> 567815 T1985 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:347)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:344)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:344)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:346)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:296)
   [junit4]   2> 
   [junit4]   2> 569313 T1981 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 569315 T1958 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1342252620
   [junit4]   2> 569318 T1958 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:36560 36560
   [junit4]   2> 569995 T1968 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:36560 36560
   [junit4]   2> 569996 T1958 oas.SolrTestCaseJ4.tearDown ###Ending testChrootBootstrap
   [junit4]   2> 569999 T1958 oas.SolrTestCaseJ4.setUp ###Starting testWithUploadDir
   [junit4]   2> 570000 T1958 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 570001 T1986 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 570100 T1958 oasc.ZkTestServer.run start zk server on port:59606
   [junit4]   2> 570101 T1958 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 570102 T1958 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 570105 T1992 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3e03f5ba name:ZooKeeperConnection Watcher:127.0.0.1:59606 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 570105 T1958 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 570106 T1958 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 570106 T1958 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/'
   [junit4]   2> 570124 T1958 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/solr.xml
   [junit4]   2> 570131 T1958 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr
   [junit4]   2> 570131 T1958 oasc.CoreContainer.<init> New CoreContainer 930186540
   [junit4]   2> 570132 T1958 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/]
   [junit4]   2> 570132 T1958 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 570132 T1958 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 570132 T1958 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 570133 T1958 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 570133 T1958 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 570133 T1958 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 570133 T1958 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 570133 T1958 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 570133 T1958 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 570135 T1958 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=0&connTimeout=0&retry=false
   [junit4]   2> 570136 T1958 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 570136 T1958 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 570137 T1958 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 570137 T1958 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59606/foo/bar3
   [junit4]   2> 570137 T1958 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 570137 T1958 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 570139 T1958 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 570139 T1994 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c9c61df name:ZooKeeperConnection Watcher:127.0.0.1:59606 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 570140 T1958 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 570140 T1958 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 570140 T1958 oascc.SolrZkClient.makePath makePath: /foo/bar3
   [junit4]   2> 570144 T1958 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 570145 T1996 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7335093d name:ZooKeeperConnection Watcher:127.0.0.1:59606/foo/bar3 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 570145 T1958 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 570146 T1958 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 570148 T1958 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 570149 T1958 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 570151 T1958 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 570152 T1958 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 570154 T1958 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 570155 T1958 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/188.138.97.18:8983_solr
   [junit4]   2> 570155 T1958 oascc.SolrZkClient.makePath makePath: /live_nodes/188.138.97.18:8983_solr
   [junit4]   2> 570157 T1958 oascc.SolrZkClient.makePath makePath: /collections
   [junit4]   2> 570158 T1958 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 570159 T1958 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 570160 T1958 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 570161 T1958 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 188.138.97.18:8983_solr
   [junit4]   2> 570161 T1958 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 570162 T1958 oasc.Overseer.start Overseer (id=92452284739551234-188.138.97.18:8983_solr-n_0000000000) starting
   [junit4]   2> 570164 T1958 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 570168 T1958 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 570169 T1998 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 570172 T1958 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 570175 T1958 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 570176 T1958 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 570178 T1997 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 570179 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/spellings.txt
   [junit4]   2> 570183 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/stopwords.txt
   [junit4]   2> 570185 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/scripts.conf
   [junit4]   2> 570187 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/admin-extra.html
   [junit4]   2> 570188 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/schema.xml
   [junit4]   2> 570190 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/solrconfig.xml
   [junit4]   2> 570193 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/admin-extra.menu-top.html
   [junit4]   2> 570194 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/elevate.xml
   [junit4]   2> 570196 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/_schema_analysis_synonyms_english.json
   [junit4]   2> 570198 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_nl.txt
   [junit4]   2> 570200 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_eu.txt
   [junit4]   2> 570202 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_ga.txt
   [junit4]   2> 570204 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_fr.txt
   [junit4]   2> 570205 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_bg.txt
   [junit4]   2> 570207 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stemdict_nl.txt
   [junit4]   2> 570209 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/contractions_ca.txt
   [junit4]   2> 570211 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_ja.txt
   [junit4]   2> 570213 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_id.txt
   [junit4]   2> 570215 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_it.txt
   [junit4]   2> 570216 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_fi.txt
   [junit4]   2> 570218 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/contractions_fr.txt
   [junit4]   2> 570220 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_el.txt
   [junit4]   2> 570222 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_en.txt
   [junit4]   2> 570224 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_da.txt
   [junit4]   2> 570225 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stoptags_ja.txt
   [junit4]   2> 570228 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_no.txt
   [junit4]   2> 570230 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_pt.txt
   [junit4]   2> 570232 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/hyphenations_ga.txt
   [junit4]   2> 570234 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_ru.txt
   [junit4]   2> 570237 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_ro.txt
   [junit4]   2> 570241 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_cz.txt
   [junit4]   2> 570243 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_hu.txt
   [junit4]   2> 570245 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_th.txt
   [junit4]   2> 570247 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_ckb.txt
   [junit4]   2> 570248 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_gl.txt
   [junit4]   2> 570250 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_ar.txt
   [junit4]   2> 570252 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_tr.txt
   [junit4]   2> 570253 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_es.txt
   [junit4]   2> 570255 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/userdict_ja.txt
   [junit4]   2> 570256 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/contractions_it.txt
   [junit4]   2> 570258 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_ca.txt
   [junit4]   2> 570260 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_lv.txt
   [junit4]   2> 570261 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_de.txt
   [junit4]   2> 570263 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/contractions_ga.txt
   [junit4]   2> 570265 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_hi.txt
   [junit4]   2> 570266 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_hy.txt
   [junit4]   2> 570268 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_sv.txt
   [junit4]   2> 570269 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_fa.txt
   [junit4]   2> 570271 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/synonyms.txt
   [junit4]   2> 570272 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/update-script.js
   [junit4]   2> 570274 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/facet_pivot.vm
   [junit4]   2> 570276 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/jquery.autocomplete.css
   [junit4]   2> 570278 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/debug.vm
   [junit4]   2> 570279 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/query_form.vm
   [junit4]   2> 570281 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/pagination_top.vm
   [junit4]   2> 570283 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/did_you_mean.vm
   [junit4]   2> 570284 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/query_spatial.vm
   [junit4]   2> 570286 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/hit_plain.vm
   [junit4]   2> 570287 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/pagination_bottom.vm
   [junit4]   2> 570289 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/header.vm
   [junit4]   2> 570290 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/layout.vm
   [junit4]   2> 570291 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/suggest.vm
   [junit4]   2> 570293 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/README.txt
   [junit4]   2> 570294 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/head.vm
   [junit4]   2> 570296 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/footer.vm
   [junit4]   2> 570297 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/results_list.vm
   [junit4]   2> 570298 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/jquery.autocomplete.js
   [junit4]   2> 570300 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/hit_grouped.vm
   [junit4]   2> 570301 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/facet_fields.vm
   [junit4]   2> 570303 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/query_group.vm
   [junit4]   2> 570305 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/query.vm
   [junit4]   2> 570306 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/cluster_results.vm
   [junit4]   2> 570308 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/VM_global_library.vm
   [junit4]   2> 570309 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/facet_queries.vm
   [junit4]   2> 570311 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/mime_type_lists.vm
   [junit4]   2> 570312 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/error.vm
   [junit4]   2> 570314 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/product_doc.vm
   [junit4]   2> 570316 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/main.css
   [junit4]   2> 570318 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/browse.vm
   [junit4]   2> 570320 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/richtext_doc.vm
   [junit4]   2> 570323 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/cluster.vm
   [junit4]   2> 570326 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/facets.vm
   [junit4]   2> 570329 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/tabs.vm
   [junit4]   2> 570331 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/join_doc.vm
   [junit4]   2> 570333 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/hit.vm
   [junit4]   2> 570334 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/facet_ranges.vm
   [junit4]   2> 570336 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/mapping-ISOLatin1Accent.txt
   [junit4]   2> 570337 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/_schema_analysis_stopwords_english.json
   [junit4]   2> 570339 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/protwords.txt
   [junit4]   2> 570340 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/_rest_managed.json
   [junit4]   2> 570341 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/xslt/example.xsl
   [junit4]   2> 570343 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/xslt/example_rss.xsl
   [junit4]   2> 570345 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/xslt/updateXml.xsl
   [junit4]   2> 570346 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/xslt/luke.xsl
   [junit4]   2> 570348 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/xslt/example_atom.xsl
   [junit4]   2> 570350 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/mapping-FoldToASCII.txt
   [junit4]   2> 570352 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/clustering/carrot2/stc-attributes.xml
   [junit4]   2> 570355 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/clustering/carrot2/lingo-attributes.xml
   [junit4]   2> 570357 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/clustering/carrot2/kmeans-attributes.xml
   [junit4]   2> 570359 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/admin-extra.menu-bottom.html
   [junit4]   2> 570361 T1958 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/currency.xml
   [junit4]   2> 570363 T1958 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr
   [junit4]   2> 570366 T1958 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/
   [junit4]   2> 570367 T1958 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 570368 T2000 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 570368 T2000 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 570369 T2000 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 570369 T2001 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 570370 T1997 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://188.138.97.18:8983/solr",
   [junit4]   2> 	  "node_name":"188.138.97.18:8983_solr",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 570370 T1997 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 570372 T2001 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 571369 T2000 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 571369 T2000 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 571370 T2000 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:collection1
   [junit4]   2> 571370 T2000 oasc.ZkController.createCollectionZkNode Setting config for collection:collection1 to testWithUploadDir
   [junit4]   2> 571370 T2000 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 571372 T2000 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 571372 T2000 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=testWithUploadDir specified config exists in ZooKeeper
   [junit4]   2> 571373 T2000 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/'
   [junit4]   2> 571388 T2000 oasc.SolrConfig.initLibs Adding specified lib dirs to ClassLoader
   [junit4]   2> 571388 T2000 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/extraction/lib (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/extraction/lib).
   [junit4]   2> 571388 T2000 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 571388 T2000 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/clustering/lib/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/clustering/lib).
   [junit4]   2> 571388 T2000 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 571389 T2000 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/langid/lib/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/langid/lib).
   [junit4]   2> 571389 T2000 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 571389 T2000 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/velocity/lib (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/velocity/lib).
   [junit4]   2> 571389 T2000 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 571404 T2000 oasu.SolrIndexConfig.<init> IndexWriter infoStream solr logging is enabled
   [junit4]   2> 571406 T2000 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 571436 T2000 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 571437 T2000 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 571440 T2000 oass.IndexSchema.readSchema [collection1] Schema name=example
   [junit4]   2> 571604 T2000 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 571606 T2000 oasr.RestManager$Registry.registerManagedResource Registered ManagedResource impl org.apache.solr.rest.schema.analysis.ManagedWordSetResource for path /schema/analysis/stopwords/english
   [junit4]   2> 571606 T2000 oasr.RestManager$Registry.registerManagedResource Registered ManagedResource impl org.apache.solr.rest.schema.analysis.ManagedSynonymFilterFactory$SynonymManager for path /schema/analysis/synonyms/english
   [junit4]   2> 571617 T2000 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 571619 T2000 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 571646 T2000 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 571647 T2000 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 571647 T2000 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data/
   [junit4]   2> 571647 T2000 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7de237b8
   [junit4]   2> 571648 T2000 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for newSearcher: org.apache.solr.core.QuerySenderListener{queries=[]}
   [junit4]   2> 571648 T2000 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for firstSearcher: org.apache.solr.core.QuerySenderListener{queries=[{q=static firstSearcher warming in solrconfig.xml}]}
   [junit4]   2> 571649 T2000 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@29e1c287
   [junit4]   2> 571649 T2000 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 571649 T2000 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 571649 T2000 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 571649 T2000 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 0 directories currently being tracked
   [junit4]   2> 571650 T2000 oasc.CoreContainer.create ERROR Error creating core [collection1]: access denied ("java.io.FilePermission" "/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data" "write") org.apache.solr.common.SolrException: access denied ("java.io.FilePermission" "/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data" "write")
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:891)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data" "write")
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.NativeFSLockFactory.<init>(NativeFSLockFactory.java:93)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:502)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
   [junit4]   2> 	... 8 more
   [junit4]   2> 
   [junit4]   2> 571651 T1958 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=930186540
   [junit4]   2> 571652 T1958 oasc.Overseer.close Overseer (id=92452284739551234-188.138.97.18:8983_solr-n_0000000000) closing
   [junit4]   2> 571653 T1997 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 188.138.97.18:8983_solr
   [junit4]   2> 573154 T2003 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:347)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:344)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:344)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:346)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:296)
   [junit4]   2> 
   [junit4]   2> 573157 T2001 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 573159 T1958 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=930186540
   [junit4]   2> 573160 T1958 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:59606 59606
   [junit4]   2> 573984 T1986 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:59606 59606
   [junit4]   2> 573985 T1958 oas.SolrTestCaseJ4.tearDown ###Ending testWithUploadDir
   [junit4]   2> 573992 T1958 oas.SolrTestCaseJ4.setUp ###Starting testInitPathExists
   [junit4]   2> 573993 T1958 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 573993 T2004 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 574093 T1958 oasc.ZkTestServer.run start zk server on port:46752
   [junit4]   2> 574094 T1958 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 574098 T1958 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 574101 T2010 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@55270fe9 name:ZooKeeperConnection Watcher:127.0.0.1:46752 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 574101 T1958 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 574101 T1958 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 574101 T1958 oascc.SolrZkClient.makePath makePath: /foo/bar4
   [junit4]   2> 574104 T1958 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/'
   [junit4]   2> 574116 T1958 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/solr.xml
   [junit4]   2> 574123 T1958 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr
   [junit4]   2> 574124 T1958 oasc.CoreContainer.<init> New CoreContainer 1903170647
   [junit4]   2> 574124 T1958 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/]
   [junit4]   2> 574124 T1958 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 574125 T1958 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 574125 T1958 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 574125 T1958 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 574126 T1958 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 574126 T1958 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 574126 T1958 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 574126 T1958 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 574127 T1958 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 574128 T1958 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=0&connTimeout=0&retry=false
   [junit4]   2> 574130 T1958 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 574130 T1958 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 574130 T1958 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 574131 T1958 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46752/foo/bar4
   [junit4]   2> 574131 T1958 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 574131 T1958 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 574141 T1958 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 574141 T2012 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9168b55 name:ZooKeeperConnection Watcher:127.0.0.1:46752 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 574143 T1958 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 574143 T1958 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 574154 T1958 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 574155 T2014 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@34b19c1c name:ZooKeeperConnection Watcher:127.0.0.1:46752/foo/bar4 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 574155 T1958 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 574156 T1958 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 574158 T1958 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 574160 T1958 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 574162 T1958 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 574163 T1958 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 574165 T1958 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 574168 T1958 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/188.138.97.18:8983_solr
   [junit4]   2> 574169 T1958 oascc.SolrZkClient.makePath makePath: /live_nodes/188.138.97.18:8983_solr
   [junit4]   2> 574170 T1958 oascc.SolrZkClient.

[...truncated too long message...]

 uncaught exception in thread: Thread[id=7467, name=Thread-2786, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path18/8/1" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([3D70C21FACD02AF3]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #11: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=7468, name=Thread-2787, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path18/8/1" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([3D70C21FACD02AF3]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 2464927 T7465 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=sq, timezone=Africa/Douala
   [junit4]   2> NOTE: Linux 3.13.0-35-generic amd64/Oracle Corporation 1.8.0_20 (64-bit)/cpus=8,threads=1,free=76306592,total=365232128
   [junit4]   2> NOTE: All tests run in this JVM: [TestFoldingMultitermQuery, StatelessScriptUpdateProcessorFactoryTest, TestSolr4Spatial2, RecoveryZkTest, TestExceedMaxTermLength, RequiredFieldsTest, TestCloudPivotFacet, TestPartialUpdateDeduplication, TestDistributedSearch, XsltUpdateRequestHandlerTest, SignatureUpdateProcessorFactoryTest, SharedFSAutoReplicaFailoverUtilsTest, TestFuzzyAnalyzedSuggestions, SystemInfoHandlerTest, TestNonNRTOpen, TestManagedSchemaFieldTypeResource, TestSystemIdResolver, TimeZoneUtilsTest, RollingRestartTest, LeaderElectionTest, TestParamSet, TestSolrDeletionPolicy1, TestSolrDeletionPolicy2, TestHashPartitioner, XmlUpdateRequestHandlerTest, HighlighterMaxOffsetTest, DistributedFacetPivotLongTailTest, UUIDUpdateProcessorFallbackTest, DistributedFacetPivotLargeTest, OverseerRolesTest, ShowFileRequestHandlerTest, AbstractAnalyticsFacetTest, NoFacetTest, DocumentAnalysisRequestHandlerTest, CursorPagingTest, UnloadDistributedZkTest, CoreMergeIndexesAdminHandlerTest, PluginInfoTest, TestOrdValues, TestTrie, TestDynamicFieldResource, DistributedSuggestComponentTest, SuggesterTest, ZkNodePropsTest, SolrIndexSplitterTest, TestOverriddenPrefixQueryForCustomFieldType, DistribDocExpirationUpdateProcessorTest, TestSearchPerf, BadComponentTest, ChangedSchemaMergeTest, TestLuceneMatchVersion, OpenExchangeRatesOrgProviderTest, TestHighFrequencyDictionaryFactory, DocValuesMultiTest, ChaosMonkeyNothingIsSafeTest, TestSolrXmlPersistence, TestMissingGroups, IndexSchemaTest, TestSchemaVersionResource, TestSimpleQParserPlugin, TestFastLRUCache, NoCacheHeaderTest, HdfsBasicDistributedZkTest, ExpressionTest, TestRTGBase, SpellPossibilityIteratorTest, JSONWriterTest, TestSolrJ, SpellingQueryConverterTest, ResponseHeaderTest, TestArbitraryIndexDir, TestDocumentBuilder, AddSchemaFieldsUpdateProcessorFactoryTest, TestSchemaSimilarityResource, BasicFunctionalityTest, TestReloadAndDeleteDocs, PreAnalyzedUpdateProcessorTest, ZkCLITest, EchoParamsTest, TestShortCircuitedRequests, TestSchemaResource, TestAnalyzeInfixSuggestions, ResponseLogComponentTest, PingRequestHandlerTest, DocumentBuilderTest, TestFastOutputStream, UpdateParamsTest, TestLRUCache, SuggesterTSTTest, HdfsSyncSliceTest, FileUtilsTest, FastVectorHighlighterTest, TestManagedResource, TestAnalyzedSuggestions, SpellCheckCollatorTest, TestCopyFieldCollectionResource, TestReplicationHandler, FieldFacetTest, DistributedDebugComponentTest, HighlighterTest, DeleteLastCustomShardedReplicaTest, TestConfig, TestStressVersions, TestMaxScoreQueryParser, JsonLoaderTest, BasicDistributedZkTest, SyncSliceTest, OverseerTest, ShardRoutingTest, ClusterStateUpdateTest, TestRandomFaceting, TestFaceting, TestRecovery, DistributedSpellCheckComponentTest, ZkControllerTest, TestStressReorder, TestJoin, TestRangeQuery, TestSolr4Spatial, SpellCheckComponentTest, PeerSyncTest, BadIndexSchemaTest, TestSort, OverseerCollectionProcessorTest, CurrencyFieldOpenExchangeTest, DistributedQueryElevationComponentTest, SuggesterWFSTTest, SolrCoreCheckLockOnStartupTest, TestWordDelimiterFilterFactory, TestRemoteStreaming, IndexBasedSpellCheckerTest, TestSurroundQueryParser, TestWriterPerf, TestReversedWildcardFilterFactory, RequestHandlersTest, TermVectorComponentTest, MoreLikeThisHandlerTest, RegexBoostProcessorTest, ReturnFieldsTest, MBeansHandlerTest, BinaryUpdateRequestHandlerTest, TestComponentsName, TestBinaryResponseWriter, TestLFUCache, AlternateDirectoryTest, TestQuerySenderNoQuery, SolrIndexConfigTest, MultiTermTest, SampleTest, TestDocSet, NumericFieldsTest, MinimalSchemaTest, TestElisionMultitermQuery, ExternalFileFieldSortTest, TestSolrCoreProperties, TestPostingsSolrHighlighter, DirectSolrConnectionTest, NotRequiredUniqueKeyTest, TestCodecSupport, SynonymTokenizerTest, TestXIncludeConfig, TestDFRSimilarityFactory, TestLMDirichletSimilarityFactory, TestLMJelinekMercerSimilarityFactory, TestBM25SimilarityFactory, TestDefaultSimilarityFactory, TestFastWriter, LegacyHTMLStripCharFilterTest, TestJmxMonitoredMap, PrimUtilsTest, DateFieldTest, SliceStateTest, CircularListTest, SolrTestCaseJ4Test, TestDistributedMissingSort, AbstractAnalyticsStatsTest, AliasIntegrationTest, CollectionsAPIAsyncDistributedZkTest, ConnectionManagerTest, DeleteReplicaTest, DistribCursorPagingTest, RemoteQueryErrorTest, SharedFSAutoReplicaFailoverTest, SolrXmlInZkTest, TestDistribDocBasedVersion, TestLeaderElectionZkExpiry, TestMiniSolrCloudCluster, TestRequestStatusCollectionAPI, TriLevelCompositeIdRoutingTest, VMParamsZkACLAndCredentialsProvidersTest, HdfsBasicDistributedZk2Test, HdfsChaosMonkeySafeLeaderTest, HdfsRecoveryZkTest, HdfsUnloadDistributedZkTest, HdfsWriteToMultipleCollectionsTest, StressHdfsTest, CachingDirectoryFactoryTest]
   [junit4] Completed on J0 in 8.10s, 1 test, 1 error <<< FAILURES!

[...truncated 188 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:471: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:60: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1358: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:961: There were test failures: 438 suites, 1830 tests, 3 suite-level errors, 2 errors, 45 ignored (23 assumptions)

Total time: 91 minutes 39 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_20 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_40-ea-b04) - Build # 11244 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11244/
Java: 64bit/jdk1.8.0_40-ea-b04 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

6 tests failed.
REGRESSION:  org.apache.solr.cloud.SyncSliceTest.testDistribSearch

Error Message:
shard1 is not consistent.  Got 305 from https://127.0.0.1:59389/x/collection1lastClient and got 5 from https://127.0.0.1:52886/x/collection1

Stack Trace:
java.lang.AssertionError: shard1 is not consistent.  Got 305 from https://127.0.0.1:59389/x/collection1lastClient and got 5 from https://127.0.0.1:52886/x/collection1
	at __randomizedtesting.SeedInfo.seed([E43CBE5F5AB8332E:65DA30472DE75312]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1169)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1148)
	at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:235)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestZkChroot

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=5267, name=coreLoadExecutor-2664-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=5267, name=coreLoadExecutor-2664-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
	at java.lang.Thread.getStackTrace(Thread.java:1552)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@5f448063 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@744ab372)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:699)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:696)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:712)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	... 1 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestZkChroot

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=5294, name=coreLoadExecutor-2674-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=5294, name=coreLoadExecutor-2674-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
	at java.lang.Thread.getStackTrace(Thread.java:1552)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@61f3295a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@458b5747)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:699)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:696)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:712)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	... 1 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestZkChroot

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=5310, name=coreLoadExecutor-2681-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=5310, name=coreLoadExecutor-2681-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
	at java.lang.Thread.getStackTrace(Thread.java:1552)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@3fac405c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@11ddca3f)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:699)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:696)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:712)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	... 1 more


FAILED:  org.apache.solr.core.CachingDirectoryFactoryTest.stressTest

Error Message:
Captured an uncaught exception in thread: Thread[id=2563, name=Thread-943, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=2563, name=Thread-943, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1/19" "write")
	at __randomizedtesting.SeedInfo.seed([E43CBE5F5AB8332E]:0)
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
	at java.security.AccessController.checkPermission(AccessController.java:884)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
	at java.nio.file.Files.createDirectory(Files.java:674)
	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
	at java.nio.file.Files.createDirectories(Files.java:727)
	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)


FAILED:  org.apache.solr.core.RAMDirectoryFactoryTest.test

Error Message:
access denied ("java.io.FilePermission" "/fake/path" "write")

Stack Trace:
java.security.AccessControlException: access denied ("java.io.FilePermission" "/fake/path" "write")
	at __randomizedtesting.SeedInfo.seed([E43CBE5F5AB8332E:6C688185F4445ED6]:0)
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
	at java.security.AccessController.checkPermission(AccessController.java:884)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
	at java.nio.file.Files.createDirectory(Files.java:674)
	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
	at java.nio.file.Files.createDirectories(Files.java:727)
	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
	at org.apache.solr.core.RAMDirectoryFactoryTest.dotestOpenReturnsTheSameForSamePath(RAMDirectoryFactoryTest.java:46)
	at org.apache.solr.core.RAMDirectoryFactoryTest.test(RAMDirectoryFactoryTest.java:33)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 10858 lines...]
   [junit4] Suite: org.apache.solr.core.RAMDirectoryFactoryTest
   [junit4]   2> 425701 T667 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@569cdbd0 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@563ad25c assuming 'simple'
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=RAMDirectoryFactoryTest -Dtests.method=test -Dtests.seed=E43CBE5F5AB8332E -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ro -Dtests.timezone=America/Detroit -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.01s J1 | RAMDirectoryFactoryTest.test <<<
   [junit4]    > Throwable #1: java.security.AccessControlException: access denied ("java.io.FilePermission" "/fake/path" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E43CBE5F5AB8332E:6C688185F4445ED6]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.RAMDirectoryFactoryTest.dotestOpenReturnsTheSameForSamePath(RAMDirectoryFactoryTest.java:46)
   [junit4]    > 	at org.apache.solr.core.RAMDirectoryFactoryTest.test(RAMDirectoryFactoryTest.java:33)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=DummyCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=DUMMY, chunkSize=453), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=DUMMY, chunkSize=453)), sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=ro, timezone=America/Detroit
   [junit4]   2> NOTE: Linux 3.13.0-35-generic amd64/Oracle Corporation 1.8.0_40-ea (64-bit)/cpus=8,threads=1,free=41062488,total=148750336
   [junit4]   2> NOTE: All tests run in this JVM: [QueryParsingTest, TestNoOpRegenerator, TestDistribDocBasedVersion, LoggingHandlerTest, TestFiltering, TestCollectionAPI, TestNonDefinedSimilarityFactory, ResourceLoaderTest, SuggestComponentTest, TestExtendedDismaxParser, DefaultValueUpdateProcessorTest, TestQueryTypes, SoftAutoCommitTest, TestTolerantSearch, TestIndexingPerformance, DebugComponentTest, RequestHandlersTest, TestBlendedInfixSuggestions, TestCollationField, UniqFieldsUpdateProcessorFactoryTest, CoreContainerCoreInitFailuresTest, TestWriterPerf, TestSolrQueryParserDefaultOperatorResource, TermVectorComponentTest, ClusterStateTest, PrimUtilsTest, ReplicationFactorTest, TestClassNameShortening, TestOmitPositions, TestSortingResponseWriter, TestMergePolicyConfig, TestPHPSerializedResponseWriter, TestRealTimeGet, TestJoin, DirectSolrConnectionTest, CollectionsAPIAsyncDistributedZkTest, RAMDirectoryFactoryTest]
   [junit4] Completed on J1 in 0.03s, 1 test, 1 error <<< FAILURES!

[...truncated 288 lines...]
   [junit4] Suite: org.apache.solr.core.CachingDirectoryFactoryTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.core.CachingDirectoryFactoryTest-E43CBE5F5AB8332E-001/init-core-data-001
   [junit4]   2> 958222 T2560 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 958224 T2560 oas.SolrTestCaseJ4.setUp ###Starting stressTest
   [junit4]   2> 958503 T2563 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@273998ef lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@660492d4 assuming 'simple'
   [junit4]   2> 958505 T2565 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@6c008a6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@47fd029d assuming 'simple'
   [junit4]   2> 958506 T2566 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@3e1536d4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7f283cef assuming 'simple'
   [junit4]   2> 958506 T2562 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@1489c8df lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@55b4c2e4 assuming 'simple'
   [junit4]   2> 958506 T2561 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@69bbf3c2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2886f9f9 assuming 'simple'
   [junit4]   2> 958507 T2564 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@1f6390a1 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5e5e9a9a assuming 'simple'
   [junit4]   2> 958507 T2568 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@7b2b9751 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3a169d6a assuming 'simple'
   [junit4]   2> 285 T2562 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-942,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path1/19" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([E43CBE5F5AB8332E]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 285 T2564 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-944,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path1/19" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([E43CBE5F5AB8332E]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 958508 T2567 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@61f1aaf3 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@20cca0c8 assuming 'simple'
   [junit4]   2> 285 T2565 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-945,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path1/19" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([E43CBE5F5AB8332E]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 958509 T2569 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@73a8fb20 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3295f11b assuming 'simple'
   [junit4]   2> 285 T2568 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-948,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path1/19" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([E43CBE5F5AB8332E]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 958509 T2570 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@2398bdc1 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@62a5b7fa assuming 'simple'
   [junit4]   2> 285 T2561 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-941,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path1/19" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([E43CBE5F5AB8332E]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 958510 T2571 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@791f7b2b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@38227110 assuming 'simple'
   [junit4]   2> 285 T2563 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-943,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path1/19" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([E43CBE5F5AB8332E]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 285 T2566 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-946,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path1/19" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([E43CBE5F5AB8332E]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 288 T2571 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-951,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path1/19" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([E43CBE5F5AB8332E]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 288 T2570 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-950,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path1/19" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([E43CBE5F5AB8332E]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 287 T2569 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-949,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path1/19" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([E43CBE5F5AB8332E]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 287 T2567 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-947,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path1/19" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([E43CBE5F5AB8332E]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 966245 T2578 oasc.CachingDirectoryFactory.close Closing RAMDirectoryFactory - 0 directories currently being tracked
   [junit4]   2> 966374 T2560 oas.SolrTestCaseJ4.tearDown ###Ending stressTest
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CachingDirectoryFactoryTest -Dtests.method=stressTest -Dtests.seed=E43CBE5F5AB8332E -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=iw -Dtests.timezone=Europe/Busingen -Dtests.file.encoding=UTF-8
   [junit4] ERROR   8.17s J0 | CachingDirectoryFactoryTest.stressTest <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=2563, name=Thread-943, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1/19" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E43CBE5F5AB8332E]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=2561, name=Thread-941, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1/19" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E43CBE5F5AB8332E]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=2562, name=Thread-942, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1/19" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E43CBE5F5AB8332E]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #4: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=2566, name=Thread-946, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1/19" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E43CBE5F5AB8332E]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #5: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=2565, name=Thread-945, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1/19" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E43CBE5F5AB8332E]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #6: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=2564, name=Thread-944, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1/19" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E43CBE5F5AB8332E]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #7: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=2568, name=Thread-948, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1/19" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E43CBE5F5AB8332E]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #8: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=2567, name=Thread-947, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1/19" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E43CBE5F5AB8332E]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #9: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=2569, name=Thread-949, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1/19" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E43CBE5F5AB8332E]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #10: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=2570, name=Thread-950, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1/19" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E43CBE5F5AB8332E]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #11: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=2571, name=Thread-951, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1/19" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E43CBE5F5AB8332E]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 966395 T2560 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=iw, timezone=Europe/Busingen
   [junit4]   2> NOTE: Linux 3.13.0-35-generic amd64/Oracle Corporation 1.8.0_40-ea (64-bit)/cpus=8,threads=1,free=87720184,total=244121600
   [junit4]   2> NOTE: All tests run in this JVM: [TestComponentsName, ResponseHeaderTest, TestFastWriter, DirectUpdateHandlerTest, DOMUtilTest, DistributedSpellCheckComponentTest, TestManagedStopFilterFactory, TestHashPartitioner, RequiredFieldsTest, TestDFRSimilarityFactory, DistribCursorPagingTest, OverseerTest, UUIDUpdateProcessorFallbackTest, XsltUpdateRequestHandlerTest, CoreMergeIndexesAdminHandlerTest, DirectSolrSpellCheckerTest, TestJmxIntegration, TestManagedResource, HighlighterConfigTest, TestPartialUpdateDeduplication, FileUtilsTest, ExternalCollectionsTest, StandardRequestHandlerTest, ShowFileRequestHandlerTest, TestDistributedMissingSort, DistribDocExpirationUpdateProcessorTest, TestFieldResource, DistributedDebugComponentTest, SignatureUpdateProcessorFactoryTest, EchoParamsTest, BadIndexSchemaTest, DisMaxRequestHandlerTest, TestShardHandlerFactory, TestMiniSolrCloudCluster, TestFastLRUCache, TestFoldingMultitermQuery, PluginInfoTest, HdfsBasicDistributedZk2Test, TestDefaultSimilarityFactory, ExternalFileFieldSortTest, ClusterStateUpdateTest, TestConfig, CurrencyFieldOpenExchangeTest, DistributedSuggestComponentTest, SpellingQueryConverterTest, OverseerCollectionProcessorTest, TestFreeTextSuggestions, TestSolrDeletionPolicy1, TestCloudPivotFacet, UnloadDistributedZkTest, HdfsRecoveryZkTest, ModifyConfFileTest, DistributedFacetPivotLargeTest, TestQuerySenderListener, TestSolr4Spatial2, StatelessScriptUpdateProcessorFactoryTest, TestHighFrequencyDictionaryFactory, TestOrdValues, OpenExchangeRatesOrgProviderTest, DocValuesMultiTest, BadComponentTest, TestSolrXmlPersistence, SpellCheckCollatorTest, TestFuzzyAnalyzedSuggestions, TestOverriddenPrefixQueryForCustomFieldType, DistributedQueryComponentCustomSortTest, AnalysisErrorHandlingTest, FastVectorHighlighterTest, TestRecoveryHdfs, SolrTestCaseJ4Test, TestIntervalFaceting, TestRawResponseWriter, TestSearchPerf, CachingDirectoryFactoryTest]
   [junit4] Completed on J0 in 8.19s, 1 test, 1 error <<< FAILURES!

[...truncated 224 lines...]
   [junit4] Suite: org.apache.solr.cloud.SyncSliceTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SyncSliceTest-E43CBE5F5AB8332E-001/init-core-data-001
   [junit4]   2> 1762268 T3944 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 1762269 T3944 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /x/
   [junit4]   2> 1762273 T3944 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 1762273 T3944 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1762274 T3945 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1762374 T3944 oasc.ZkTestServer.run start zk server on port:34650
   [junit4]   2> 1762375 T3944 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1762376 T3944 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1762379 T3951 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5f8df87b name:ZooKeeperConnection Watcher:127.0.0.1:34650 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1762380 T3944 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1762380 T3944 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1762380 T3944 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1762385 T3944 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1762387 T3944 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1762393 T3953 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5e21e84a name:ZooKeeperConnection Watcher:127.0.0.1:34650/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1762394 T3944 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1762394 T3944 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1762394 T3944 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1762397 T3944 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1762400 T3944 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1762402 T3944 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1762405 T3944 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1762405 T3944 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1762409 T3944 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1762410 T3944 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1762413 T3944 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1762413 T3944 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1762416 T3944 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1762417 T3944 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1762420 T3944 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1762420 T3944 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1762423 T3944 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1762423 T3944 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1762426 T3944 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1762427 T3944 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1762430 T3944 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1762430 T3944 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1762433 T3944 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1762434 T3944 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1762436 T3944 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1762437 T3944 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1762440 T3944 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1762440 T3944 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1762443 T3944 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1762444 T3944 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1762446 T3955 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3cc92289 name:ZooKeeperConnection Watcher:127.0.0.1:34650/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1762446 T3944 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1762446 T3944 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1762687 T3944 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1762689 T3944 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1762692 T3944 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:35041
   [junit4]   2> 1762694 T3944 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1762694 T3944 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1762694 T3944 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SyncSliceTest-E43CBE5F5AB8332E-001/tempDir-002
   [junit4]   2> 1762695 T3944 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SyncSliceTest-E43CBE5F5AB8332E-001/tempDir-002/'
   [junit4]   2> 1762704 T3944 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SyncSliceTest-E43CBE5F5AB8332E-001/tempDir-002/solr.xml
   [junit4]   2> 1762720 T3944 oasc.CoreContainer.<init> New CoreContainer 1008789266
   [junit4]   2> 1762720 T3944 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SyncSliceTest-E43CBE5F5AB8332E-001/tempDir-002/]
   [junit4]   2> 1762721 T3944 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1762721 T3944 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1762721 T3944 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1762721 T3944 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1762722 T3944 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1762722 T3944 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1762722 T3944 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1762722 T3944 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1762723 T3944 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1762724 T3944 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1762725 T3944 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1762725 T3944 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1762725 T3944 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1762726 T3944 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:34650/solr
   [junit4]   2> 1762726 T3944 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1762727 T3944 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1762727 T3944 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1762730 T3966 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2de28715 name:ZooKeeperConnection Watcher:127.0.0.1:34650 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1762731 T3944 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1762731 T3944 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1762733 T3944 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1762735 T3968 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4e512f49 name:ZooKeeperConnection Watcher:127.0.0.1:34650/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1762735 T3944 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1762736 T3944 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1762739 T3944 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1762742 T3944 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1762744 T3944 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1762746 T3944 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1762749 T3944 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1762751 T3944 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35041_x
   [junit4]   2> 1762752 T3944 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35041_x
   [junit4]   2> 1762754 T3944 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1762756 T3944 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1762759 T3944 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1762761 T3944 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:35041_x
   [junit4]   2> 1762761 T3944 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1762764 T3944 oasc.Overseer.start Overseer (id=92451566781857796-127.0.0.1:35041_x-n_0000000000) starting
   [junit4]   2> 1762767 T3944 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1762778 T3944 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1762778 T3970 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1762779 T3944 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1762783 T3944 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1762785 T3944 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1762790 T3969 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1762791 T3972 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 1762792 T3972 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1762793 T3972 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1762793 T3973 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1762797 T3969 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:35041/x",
   [junit4]   2> 	  "node_name":"127.0.0.1:35041_x",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1762797 T3969 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 1762797 T3969 oasc.Overseer$ClusterStateUpdater.createCollection state version control_collection 1
   [junit4]   2> 1762798 T3969 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1762801 T3973 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1763794 T3972 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1763794 T3972 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 1763795 T3972 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1763795 T3972 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1763797 T3972 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1763797 T3972 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SyncSliceTest-E43CBE5F5AB8332E-001/tempDir-002/collection1/'
   [junit4]   2> 1763798 T3972 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SyncSliceTest-E43CBE5F5AB8332E-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 1763798 T3972 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SyncSliceTest-E43CBE5F5AB8332E-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 1763798 T3972 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SyncSliceTest-E43CBE5F5AB8332E-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 1763826 T3972 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1763840 T3972 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1763842 T3972 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1763846 T3972 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1763959 T3972 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1763964 T3972 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1763965 T3972 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1763974 T3972 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1763975 T3972 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1763977 T3972 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1763977 T3972 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1763978 T3972 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1763978 T3972 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1763979 T3972 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1763979 T3972 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1763980 T3972 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1763980 T3972 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1763980 T3972 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SyncSliceTest-E43CBE5F5AB8332E-001/tempDir-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SyncSliceTest-E43CBE5F5AB8332E-001/tempDir-001/control/data/
   [junit4]   2> 1763980 T3972 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7ec39b10
   [junit4]   2> 1763980 T3972 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SyncSliceTest-E43CBE5F5AB8332E-001/tempDir-001/control/data
   [junit4]   2> 1763981 T3972 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SyncSliceTest-E43CBE5F5AB8332E-001/tempDir-001/control/data/index/
   [junit4]   2> 1763981 T3972 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SyncSliceTest-E43CBE5F5AB8332E-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1763981 T3972 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SyncSliceTest-E43CBE5F5AB8332E-001/tempDir-001/control/data/index
   [junit4]   2> 1763981 T3972 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=49, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 1763982 T3972 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@470c41ba lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7acbd738),segFN=segments_1,generation=1}
   [junit4]   2> 1763982 T3972 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1763984 T3972 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1763984 T3972 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1763984 T3972 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1763984 T3972 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1763984 T3972 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1763985 T3972 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1763985 T3972 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-

[...truncated too long message...]

stRawResponseWriter, TestSearchPerf, CachingDirectoryFactoryTest, TestReloadAndDeleteDocs, TermsComponentTest, TestSystemIdResolver, TestExceedMaxTermLength, CoreAdminHandlerTest, ResponseLogComponentTest, TestRandomDVFaceting, ShardRoutingTest, TestReplicationHandler, TestLuceneMatchVersion, TestSchemaVersionResource, TestMissingGroups, TestSort, PathHierarchyTokenizerFactoryTest, RollingRestartTest, TestSolrCoreProperties, TestSurroundQueryParser, HdfsDirectoryTest, RecoveryZkTest, TestStressUserVersions, TestChildDocTransformer, HighlighterTest, TestLRUCache, FieldFacetTest, OverseerStatusTest, OutOfBoxZkACLAndCredentialsProvidersTest, TestAnalyzedSuggestions, AssignTest, MoreLikeThisHandlerTest, PeerSyncTest, TestCustomSort, TestCopyFieldCollectionResource, TestConfigSets, WordBreakSolrSpellCheckerTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, BasicZkTest, TestRandomFaceting, TestZkChroot]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestZkChroot -Dtests.seed=E43CBE5F5AB8332E -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=cs_CZ -Dtests.timezone=America/Edmonton -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J0 | TestZkChroot (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=5267, name=coreLoadExecutor-2664-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
   [junit4]    > 	at java.lang.Thread.getStackTrace(Thread.java:1552)
   [junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
   [junit4]    > 	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
   [junit4]    > 	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
   [junit4]    > 	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
   [junit4]    > 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
   [junit4]    > 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
   [junit4]    > 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
   [junit4]    > 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@5f448063 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@744ab372)
   [junit4]    > 	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
   [junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
   [junit4]    > 	... 1 moreThrowable #2: com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=5294, name=coreLoadExecutor-2674-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
   [junit4]    > 	at java.lang.Thread.getStackTrace(Thread.java:1552)
   [junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
   [junit4]    > 	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
   [junit4]    > 	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
   [junit4]    > 	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
   [junit4]    > 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
   [junit4]    > 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
   [junit4]    > 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
   [junit4]    > 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@61f3295a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@458b5747)
   [junit4]    > 	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
   [junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
   [junit4]    > 	... 1 moreThrowable #3: com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=5310, name=coreLoadExecutor-2681-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
   [junit4]    > 	at java.lang.Thread.getStackTrace(Thread.java:1552)
   [junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
   [junit4]    > 	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
   [junit4]    > 	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
   [junit4]    > 	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
   [junit4]    > 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
   [junit4]    > 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
   [junit4]    > 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
   [junit4]    > 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@3fac405c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@11ddca3f)
   [junit4]    > 	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
   [junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
   [junit4]    > 	... 1 more
   [junit4] Completed on J0 in 13.17s, 4 tests, 3 errors <<< FAILURES!

[...truncated 659 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:471: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:60: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1358: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:961: There were test failures: 438 suites, 1830 tests, 3 suite-level errors, 2 errors, 1 failure, 45 ignored (23 assumptions)

Total time: 90 minutes 57 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_40-ea-b04 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_67) - Build # 11243 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11243/
Java: 32bit/jdk1.7.0_67 -server -XX:+UseParallelGC

6 tests failed.
REGRESSION:  org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.testDistribSearch

Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:45661, http://127.0.0.1:39262, http://127.0.0.1:46288]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:45661, http://127.0.0.1:39262, http://127.0.0.1:46288]
	at __randomizedtesting.SeedInfo.seed([7274912D883006C4:F3921F35FF6F66F8]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:322)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:874)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:658)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601)
	at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.removeAndWaitForLastReplicaGone(DeleteLastCustomShardedReplicaTest.java:116)
	at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.doTest(DeleteLastCustomShardedReplicaTest.java:106)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not  remove replica : customcollreplicadeletion/a/core_node1
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:550)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:343)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:304)
	... 46 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestZkChroot

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=848, name=coreLoadExecutor-498-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=848, name=coreLoadExecutor-498-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
	at java.lang.Thread.getStackTrace(Thread.java:1589)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@87aea4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1fbc327)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:699)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:696)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:712)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	... 1 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestZkChroot

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=866, name=coreLoadExecutor-505-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=866, name=coreLoadExecutor-505-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
	at java.lang.Thread.getStackTrace(Thread.java:1589)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@118ac99 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@cf1ecf)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:699)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:696)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:712)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	... 1 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestZkChroot

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=882, name=coreLoadExecutor-512-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=882, name=coreLoadExecutor-512-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
	at java.lang.Thread.getStackTrace(Thread.java:1589)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@4b11a5 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1efa979)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:699)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:696)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:712)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	... 1 more


FAILED:  org.apache.solr.core.CachingDirectoryFactoryTest.stressTest

Error Message:
Captured an uncaught exception in thread: Thread[id=1250, name=Thread-410, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1250, name=Thread-410, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path17" "write")
	at __randomizedtesting.SeedInfo.seed([7274912D883006C4]:0)
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
	at java.security.AccessController.checkPermission(AccessController.java:559)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
	at java.nio.file.Files.createDirectory(Files.java:630)
	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
	at java.nio.file.Files.createDirectories(Files.java:683)
	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)


FAILED:  org.apache.solr.core.RAMDirectoryFactoryTest.test

Error Message:
access denied ("java.io.FilePermission" "/fake/path" "write")

Stack Trace:
java.security.AccessControlException: access denied ("java.io.FilePermission" "/fake/path" "write")
	at __randomizedtesting.SeedInfo.seed([7274912D883006C4:FA20AEF726CC6B3C]:0)
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
	at java.security.AccessController.checkPermission(AccessController.java:559)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
	at java.nio.file.Files.createDirectory(Files.java:630)
	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
	at java.nio.file.Files.createDirectories(Files.java:683)
	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
	at org.apache.solr.core.RAMDirectoryFactoryTest.dotestOpenReturnsTheSameForSamePath(RAMDirectoryFactoryTest.java:46)
	at org.apache.solr.core.RAMDirectoryFactoryTest.test(RAMDirectoryFactoryTest.java:33)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 10739 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestZkChroot
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.TestZkChroot-7274912D883006C4-001/init-core-data-001
   [junit4]   2> 294937 T824 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 294940 T824 oas.SolrTestCaseJ4.setUp ###Starting testNoBootstrapConf
   [junit4]   2> 294940 T824 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 294941 T825 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 295041 T824 oasc.ZkTestServer.run start zk server on port:52167
   [junit4]   2> 295042 T824 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 295043 T824 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 295045 T831 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15ff96b name:ZooKeeperConnection Watcher:127.0.0.1:52167 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 295045 T824 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 295046 T824 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 295047 T824 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/'
   [junit4]   2> 295063 T824 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/solr.xml
   [junit4]   2> 295078 T824 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr
   [junit4]   2> 295078 T824 oasc.CoreContainer.<init> New CoreContainer 7379216
   [junit4]   2> 295079 T824 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/]
   [junit4]   2> 295079 T824 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 295080 T824 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 295080 T824 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 295080 T824 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 295080 T824 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 295080 T824 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 295081 T824 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 295081 T824 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 295081 T824 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 295082 T824 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=0&connTimeout=0&retry=false
   [junit4]   2> 295082 T824 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 295082 T824 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 295082 T824 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 295083 T824 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52167/foo/bar2
   [junit4]   2> 295083 T824 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 295083 T824 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 295084 T824 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 295085 T833 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5114bf name:ZooKeeperConnection Watcher:127.0.0.1:52167 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 295085 T824 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 295086 T824 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 295089 T824 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:52167 52167
   [junit4]   2> 295647 T825 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:52167 52167
   [junit4]   2> 295649 T824 oas.SolrTestCaseJ4.tearDown ###Ending testNoBootstrapConf
   [junit4]   2> 295657 T824 oas.SolrTestCaseJ4.setUp ###Starting testWithUploadDir
   [junit4]   2> 295657 T824 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 295658 T834 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 295758 T824 oasc.ZkTestServer.run start zk server on port:43022
   [junit4]   2> 295759 T824 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 295760 T824 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 295763 T840 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1aebb45 name:ZooKeeperConnection Watcher:127.0.0.1:43022 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 295763 T824 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 295764 T824 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 295765 T824 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/'
   [junit4]   2> 295781 T824 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/solr.xml
   [junit4]   2> 295799 T824 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr
   [junit4]   2> 295799 T824 oasc.CoreContainer.<init> New CoreContainer 9179682
   [junit4]   2> 295799 T824 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/]
   [junit4]   2> 295800 T824 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 295800 T824 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 295801 T824 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 295801 T824 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 295801 T824 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 295802 T824 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 295802 T824 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 295802 T824 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 295802 T824 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 295803 T824 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=0&connTimeout=0&retry=false
   [junit4]   2> 295803 T824 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 295803 T824 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 295804 T824 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 295804 T824 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43022/foo/bar3
   [junit4]   2> 295804 T824 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 295805 T824 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 295805 T824 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 295807 T842 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fab402 name:ZooKeeperConnection Watcher:127.0.0.1:43022 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 295807 T824 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 295808 T824 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 295809 T824 oascc.SolrZkClient.makePath makePath: /foo/bar3
   [junit4]   2> 295816 T824 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 295817 T844 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d2eb50 name:ZooKeeperConnection Watcher:127.0.0.1:43022/foo/bar3 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 295818 T824 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 295820 T824 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 295824 T824 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 295827 T824 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 295831 T824 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 295835 T824 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 295838 T824 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 295840 T824 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/188.138.97.18:8983_solr
   [junit4]   2> 295841 T824 oascc.SolrZkClient.makePath makePath: /live_nodes/188.138.97.18:8983_solr
   [junit4]   2> 295844 T824 oascc.SolrZkClient.makePath makePath: /collections
   [junit4]   2> 295846 T824 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 295848 T824 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 295851 T824 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 295853 T824 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 188.138.97.18:8983_solr
   [junit4]   2> 295854 T824 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 295856 T824 oasc.Overseer.start Overseer (id=92450710206808066-188.138.97.18:8983_solr-n_0000000000) starting
   [junit4]   2> 295859 T824 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 295866 T824 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 295867 T846 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 295868 T824 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 295871 T824 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 295873 T824 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 295877 T845 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 295878 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/spellings.txt
   [junit4]   2> 295883 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/stopwords.txt
   [junit4]   2> 295885 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/scripts.conf
   [junit4]   2> 295887 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/admin-extra.html
   [junit4]   2> 295890 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/schema.xml
   [junit4]   2> 295894 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/solrconfig.xml
   [junit4]   2> 295899 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/admin-extra.menu-top.html
   [junit4]   2> 295902 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/elevate.xml
   [junit4]   2> 295905 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/_schema_analysis_synonyms_english.json
   [junit4]   2> 295909 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_nl.txt
   [junit4]   2> 295913 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_eu.txt
   [junit4]   2> 295917 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_ga.txt
   [junit4]   2> 295921 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_fr.txt
   [junit4]   2> 295925 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_bg.txt
   [junit4]   2> 295930 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stemdict_nl.txt
   [junit4]   2> 295934 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/contractions_ca.txt
   [junit4]   2> 295937 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_ja.txt
   [junit4]   2> 295941 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_id.txt
   [junit4]   2> 295944 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_it.txt
   [junit4]   2> 295948 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_fi.txt
   [junit4]   2> 295952 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/contractions_fr.txt
   [junit4]   2> 295956 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_el.txt
   [junit4]   2> 295959 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_en.txt
   [junit4]   2> 295963 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_da.txt
   [junit4]   2> 295967 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stoptags_ja.txt
   [junit4]   2> 295971 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_no.txt
   [junit4]   2> 295975 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_pt.txt
   [junit4]   2> 295978 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/hyphenations_ga.txt
   [junit4]   2> 295981 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_ru.txt
   [junit4]   2> 295985 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_ro.txt
   [junit4]   2> 295987 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_cz.txt
   [junit4]   2> 295989 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_hu.txt
   [junit4]   2> 295992 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_th.txt
   [junit4]   2> 295997 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_ckb.txt
   [junit4]   2> 296000 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_gl.txt
   [junit4]   2> 296004 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_ar.txt
   [junit4]   2> 296007 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_tr.txt
   [junit4]   2> 296010 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_es.txt
   [junit4]   2> 296014 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/userdict_ja.txt
   [junit4]   2> 296018 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/contractions_it.txt
   [junit4]   2> 296022 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_ca.txt
   [junit4]   2> 296025 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_lv.txt
   [junit4]   2> 296029 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_de.txt
   [junit4]   2> 296032 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/contractions_ga.txt
   [junit4]   2> 296035 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_hi.txt
   [junit4]   2> 296038 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_hy.txt
   [junit4]   2> 296042 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_sv.txt
   [junit4]   2> 296045 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/lang/stopwords_fa.txt
   [junit4]   2> 296048 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/synonyms.txt
   [junit4]   2> 296052 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/update-script.js
   [junit4]   2> 296056 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/facet_pivot.vm
   [junit4]   2> 296060 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/jquery.autocomplete.css
   [junit4]   2> 296063 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/debug.vm
   [junit4]   2> 296066 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/query_form.vm
   [junit4]   2> 296068 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/pagination_top.vm
   [junit4]   2> 296070 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/did_you_mean.vm
   [junit4]   2> 296073 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/query_spatial.vm
   [junit4]   2> 296075 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/hit_plain.vm
   [junit4]   2> 296078 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/pagination_bottom.vm
   [junit4]   2> 296080 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/header.vm
   [junit4]   2> 296082 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/layout.vm
   [junit4]   2> 296085 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/suggest.vm
   [junit4]   2> 296087 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/README.txt
   [junit4]   2> 296090 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/head.vm
   [junit4]   2> 296092 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/footer.vm
   [junit4]   2> 296095 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/results_list.vm
   [junit4]   2> 296098 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/jquery.autocomplete.js
   [junit4]   2> 296102 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/hit_grouped.vm
   [junit4]   2> 296106 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/facet_fields.vm
   [junit4]   2> 296109 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/query_group.vm
   [junit4]   2> 296113 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/query.vm
   [junit4]   2> 296116 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/cluster_results.vm
   [junit4]   2> 296120 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/VM_global_library.vm
   [junit4]   2> 296123 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/facet_queries.vm
   [junit4]   2> 296126 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/mime_type_lists.vm
   [junit4]   2> 296128 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/error.vm
   [junit4]   2> 296130 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/product_doc.vm
   [junit4]   2> 296132 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/main.css
   [junit4]   2> 296134 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/browse.vm
   [junit4]   2> 296137 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/richtext_doc.vm
   [junit4]   2> 296139 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/cluster.vm
   [junit4]   2> 296141 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/facets.vm
   [junit4]   2> 296143 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/tabs.vm
   [junit4]   2> 296147 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/join_doc.vm
   [junit4]   2> 296149 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/hit.vm
   [junit4]   2> 296151 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/velocity/facet_ranges.vm
   [junit4]   2> 296154 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/mapping-ISOLatin1Accent.txt
   [junit4]   2> 296156 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/_schema_analysis_stopwords_english.json
   [junit4]   2> 296159 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/protwords.txt
   [junit4]   2> 296162 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/_rest_managed.json
   [junit4]   2> 296165 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/xslt/example.xsl
   [junit4]   2> 296169 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/xslt/example_rss.xsl
   [junit4]   2> 296172 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/xslt/updateXml.xsl
   [junit4]   2> 296176 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/xslt/luke.xsl
   [junit4]   2> 296179 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/xslt/example_atom.xsl
   [junit4]   2> 296182 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/mapping-FoldToASCII.txt
   [junit4]   2> 296186 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/clustering/carrot2/stc-attributes.xml
   [junit4]   2> 296191 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/clustering/carrot2/lingo-attributes.xml
   [junit4]   2> 296195 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/clustering/carrot2/kmeans-attributes.xml
   [junit4]   2> 296199 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/admin-extra.menu-bottom.html
   [junit4]   2> 296202 T824 oascc.SolrZkClient.makePath makePath: /configs/testWithUploadDir/currency.xml
   [junit4]   2> 296207 T824 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr
   [junit4]   2> 296212 T824 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/
   [junit4]   2> 296213 T824 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 296214 T848 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 296214 T848 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 296215 T848 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 296216 T849 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 296219 T845 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "node_name":"188.138.97.18:8983_solr",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://188.138.97.18:8983/solr",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 296219 T845 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 296223 T849 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 297216 T848 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 297216 T848 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 297217 T848 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:collection1
   [junit4]   2> 297218 T848 oasc.ZkController.createCollectionZkNode Setting config for collection:collection1 to testWithUploadDir
   [junit4]   2> 297218 T848 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 297221 T848 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 297223 T848 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=testWithUploadDir specified config exists in ZooKeeper
   [junit4]   2> 297223 T848 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/'
   [junit4]   2> 297258 T848 oasc.SolrConfig.initLibs Adding specified lib dirs to ClassLoader
   [junit4]   2> 297260 T848 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/extraction/lib (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/extraction/lib).
   [junit4]   2> 297261 T848 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 297262 T848 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/clustering/lib/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/clustering/lib).
   [junit4]   2> 297262 T848 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 297263 T848 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/langid/lib/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/langid/lib).
   [junit4]   2> 297263 T848 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 297264 T848 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/velocity/lib (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/velocity/lib).
   [junit4]   2> 297265 T848 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 297294 T848 oasu.SolrIndexConfig.<init> IndexWriter infoStream solr logging is enabled
   [junit4]   2> 297297 T848 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 297351 T848 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 297352 T848 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 297360 T848 oass.IndexSchema.readSchema [collection1] Schema name=example
   [junit4]   2> 297439 T3 oasc.CoreContainer.finalize ERROR CoreContainer was not close prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!  instance=7379216
   [junit4]   2> 297687 T848 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 297688 T848 oasr.RestManager$Registry.registerManagedResource Registered ManagedResource impl org.apache.solr.rest.schema.analysis.ManagedWordSetResource for path /schema/analysis/stopwords/english
   [junit4]   2> 297688 T848 oasr.RestManager$Registry.registerManagedResource Registered ManagedResource impl org.apache.solr.rest.schema.analysis.ManagedSynonymFilterFactory$SynonymManager for path /schema/analysis/synonyms/english
   [junit4]   2> 297713 T848 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 297722 T848 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 297836 T848 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 297836 T848 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 297837 T848 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data/
   [junit4]   2> 297837 T848 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1195d5
   [junit4]   2> 297845 T848 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for newSearcher: org.apache.solr.core.QuerySenderListener{queries=[]}
   [junit4]   2> 297845 T848 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for firstSearcher: org.apache.solr.core.QuerySenderListener{queries=[{q=static firstSearcher warming in solrconfig.xml}]}
   [junit4]   2> 297852 T848 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@36d511
   [junit4]   2> 297853 T848 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 297853 T848 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 297853 T848 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 297853 T848 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 0 directories currently being tracked
   [junit4]   2> 297854 T848 oasc.CoreContainer.create ERROR Error creating core [collection1]: access denied ("java.io.FilePermission" "/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data" "write") org.apache.solr.common.SolrException: access denied ("java.io.FilePermission" "/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data" "write")
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:890)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data" "write")
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.NativeFSLockFactory.<init>(NativeFSLockFactory.java:93)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:502)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
   [junit4]   2> 	... 8 more
   [junit4]   2> 
   [junit4]   2> 297856 T824 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=9179682
   [junit4]   2> 297857 T824 oasc.Overseer.close Overseer (id=92450710206808066-188.138.97.18:8983_solr-n_0000000000) closing
   [junit4]   2> 297858 T845 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 188.138.97.18:8983_solr
   [junit4]   2> 297861 T849 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 297861 T851 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:347)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:344)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:344)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:346)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:296)
   [junit4]   2> 
   [junit4]   2> 297863 T824 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=9179682
   [junit4]   2> 297865 T824 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:43022 43022
   [junit4]   2> 298418 T834 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:43022 43022
   [junit4]   2> 298420 T824 oas.SolrTestCaseJ4.tearDown ###Ending testWithUploadDir
   [junit4]   2> 298428 T824 oas.SolrTestCaseJ4.setUp ###Starting testInitPathExists
   [junit4]   2> 298429 T824 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 298430 T852 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 298530 T824 oasc.ZkTestServer.run start zk server on port:41739
   [junit4]   2> 298531 T824 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 298532 T824 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 298534 T858 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c4c613 name:ZooKeeperConnection Watcher:127.0.0.1:41739 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 298534 T824 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 298535 T824 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 298535 T824 oascc.SolrZkClient.makePath makePath: /foo/bar4
   [junit4]   2> 298541 T824 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/'
   [junit4]   2> 298571 T824 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/solr.xml
   [junit4]   2> 298589 T824 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr
   [junit4]   2> 298590 T824 oasc.CoreContainer.<init> New CoreContainer 29574453
   [junit4]   2> 298590 T824 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/]
   [junit4]   2> 298591 T824 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 298591 T824 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 298591 T824 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 298591 T824 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 298592 T824 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 298592 T824 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 298592 T824 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 298593 T824 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 298593 T824 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 298593 T824 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=0&connTimeout=0&retry=false
   [junit4]   2> 298594 T824 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 298594 T824 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 298594 T824 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 298594 T824 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41739/foo/bar4
   [junit4]   2> 298595 T824 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 298595 T824 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 298596 T824 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 298597 T860 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f4a7c9 name:ZooKeeperConnection Watcher:127.0.0.1:41739 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 298597 T824 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 298598 T824 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 298601 T824 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 298602 T862 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15103e7 name:ZooKeeperConnection Watcher:127.0.0.1:41739/foo/bar4 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 298602 T824 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 298603 T824 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 298607 T824 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 298610 T824 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 298612 T824 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 298614 T824 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 298617 T824 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 298618 T824 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/188.138.97.18:8983_solr
   [junit4]   2> 298619 T824 oascc.SolrZkClient.makePath makePath: /live_nodes/188.138.97.18:8983_solr
   [junit4]   2> 298621 T824 oascc.SolrZkClient.makePath makePath: /collections
   [junit4]   2> 298622 T824 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 298624 T824 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 298626 T824 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 298628 T824 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 188.138.97.18:8983_solr
   [junit4]   2> 298628 T824 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 298630 T824 oasc.Overseer.start Overseer (id=92450710388408322-188.138.97.18:8983_solr-n_0000000000) starting
   [junit4]   2> 298633 T824 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 298641 T824 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 298641 T864 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 298642 T824 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 298645 T824 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 298647 T824 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 298650 T824 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr
   [junit4]   2> 298651 T863 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 298654 T824 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/
   [junit4]   2> 298655 T824 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 298656 T824 oasc.ZkController.bootstrapConf bootstrapping config for 1 cores into ZooKeeper using solr.xml from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/
   [junit4]   2> 298656 T824 oasc.ZkController.bootstrapConf Uploading directory /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/conf with name collection1 for SolrCore collection1
   [junit4]   2> 298657 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/spellings.txt
   [junit4]   2> 298662 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/stopwords.txt
   [junit4]   2> 298664 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/scripts.conf
   [junit4]   2> 298666 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/admin-extra.html
   [junit4]   2> 298668 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/schema.xml
   [junit4]   2> 298670 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/solrconfig.xml
   [junit4]   2> 298672 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/admin-extra.menu-top.html
   [junit4]   2> 298674 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/elevate.xml
   [junit4]   2> 298675 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/_schema_analysis_synonyms_english.json
   [junit4]   2> 298677 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_nl.txt
   [junit4]   2> 298679 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_eu.txt
   [junit4]   2> 298681 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ga.txt
   [junit4]   2> 298682 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_fr.txt
   [junit4]   2> 298684 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_bg.txt
   [junit4]   2> 298686 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stemdict_nl.txt
   [junit4]   2> 298688 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/contractions_ca.txt
   [junit4]   2> 298691 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ja.txt
   [junit4]   2> 298693 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_id.txt
   [junit4]   2> 298696 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_it.txt
   [junit4]   2> 298699 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_fi.txt
   [junit4]   2> 298702 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/contractions_fr.txt
   [junit4]   2> 298705 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_el.txt
   [junit4]   2> 298709 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_en.txt
   [junit4]   2> 298712 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_da.txt
   [junit4]   2> 298715 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stoptags_ja.txt
   [junit4]   2> 298718 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_no.txt
   [junit4]   2> 298729 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_pt.txt
   [junit4]   2> 298732 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/hyphenations_ga.txt
   [junit4]   2> 298735 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ru.txt
   [junit4]   2> 298737 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ro.txt
   [junit4]   2> 298739 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_cz.txt
   [junit4]   2> 298741 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_hu.txt
   [junit4]   2> 298743 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_th.txt
   [junit4]   2> 298745 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ckb.txt
   [junit4]   2> 298747 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_gl.txt
   [junit4]   2> 298749 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ar.txt
   [junit4]   2> 298751 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_tr.txt
   [junit4]   2> 298753 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_es.txt
   [junit4]   2> 298756 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/userdict_ja.txt
   [junit4]   2> 298758 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/contractions_it.txt
   [junit4]   2> 298761 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ca.txt
   [junit4]   2> 298764 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_lv.txt
   [junit4]   2> 298765 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_de.txt
   [junit4]   2> 298767 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/contractions_ga.txt
   [junit4]   2> 298769 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_hi.txt
   [junit4]   2> 298771 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_hy.txt
   [junit4]   2> 298773 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_sv.txt
   [junit4]   2> 298775 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_fa.txt
   [junit4]   2> 298777 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/synonyms.txt
   [junit4]   2> 298779 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/update-script.js
   [junit4]   2> 298782 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facet_pivot.vm
   [junit4]   2> 298786 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/jquery.autocomplete.css
   [junit4]   2> 298788 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/debug.vm
   [junit4]   2> 298790 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/query_form.vm
   [junit4]   2> 298792 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/pagination_top.vm
   [junit4]   2> 298794 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/did_you_mean.vm
   [junit4]   2> 298796 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/query_spatial.vm
   [junit4]   2> 298798 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/hit_plain.vm
   [junit4]   2> 298800 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/pagination_bottom.vm
   [junit4]   2> 298802 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/header.vm
   [junit4]   2> 298804 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/layout.vm
   [junit4]   2> 298807 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/suggest.vm
   [junit4]   2> 298809 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/README.txt
   [junit4]   2> 298812 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/head.vm
   [junit4]   2> 298815 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/footer.vm
   [junit4]   2> 298818 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/results_list.vm
   [junit4]   2> 298821 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/jquery.autocomplete.js
   [junit4]   2> 298823 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/hit_grouped.vm
   [junit4]   2> 298825 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facet_fields.vm
   [junit4]   2> 298828 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/query_group.vm
   [junit4]   2> 298830 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/query.vm
   [junit4]   2> 298834 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/cluster_results.vm
   [junit4]   2> 298837 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/VM_global_library.vm
   [junit4]   2> 298839 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facet_queries.vm
   [junit4]   2> 298841 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/mime_type_lists.vm
   [junit4]   2> 298844 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/error.vm
   [junit4]   2> 298848 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/product_doc.vm
   [junit4]   2> 298851 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/main.css
   [junit4]   2> 298854 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/browse.vm
   [junit4]   2> 298858 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/richtext_doc.vm
   [junit4]   2> 298862 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/cluster.vm
   [junit4]   2> 298865 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facets.vm
   [junit4]   2> 298868 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/tabs.vm
   [junit4]   2> 298872 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/join_doc.vm
   [junit4]   2> 298876 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/hit.vm
   [junit4]   2> 298879 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facet_ranges.vm
   [junit4]   2> 298883 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 298885 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/_schema_analysis_stopwords_english.json
   [junit4]   2> 298888 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/protwords.txt
   [junit4]   2> 298891 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/_rest_managed.json
   [junit4]   2> 298893 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/example.xsl
   [junit4]   2> 298896 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/example_rss.xsl
   [junit4]   2> 298899 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/updateXml.xsl
   [junit4]   2> 298903 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/luke.xsl
   [junit4]   2> 298906 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/example_atom.xsl
   [junit4]   2> 298909 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/mapping-FoldToASCII.txt
   [junit4]   2> 298913 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/clustering/carrot2/stc-attributes.xml
   [junit4]   2> 298918 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/clustering/carrot2/lingo-attributes.xml
   [junit4]   2> 298921 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/clustering/carrot2/kmeans-attributes.xml
   [junit4]   2> 298924 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/admin-extra.menu-bottom.html
   [junit4]   2> 298926 T824 oascc.SolrZkClient.makePath makePath: /configs/collection1/currency.xml
   [junit4]   2> 298929 T824 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr
   [junit4]   2> 298931 T824 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/
   [junit4]   2> 298931 T824 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 298932 T866 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 298932 T866 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 298933 T866 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 298933 T867 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 298935 T863 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "node_name":"188.138.97.18:8983_solr",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://188.138.97.18:8983/solr",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 298935 T863 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 298938 T867 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 299933 T866 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 299934 T866 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 299934 T866 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:collection1
   [junit4]   2> 299935 T866 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 299937 T866 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 299937 T866 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=collection1 specified config exists in ZooKeeper
   [junit4]   2> 299938 T866 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/'
   [junit4]   2> 299973 T866 oasc.SolrConfig.initLibs Adding specified lib dirs to ClassLoader
   [junit4]   2> 299974 T866 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/extraction/lib (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/extraction/lib).
   [junit4]   2> 299974 T866 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 299974 T866 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/clustering/lib/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/clustering/lib).
   [junit4]   2> 299974 T866 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 299975 T866 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/langid/lib/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/langid/lib).
   [junit4]   2> 299975 T866 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 299975 T866 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/velocity/lib (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/velocity/lib).
   [junit4]   2> 299975 T866 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 300003 T866 oasu.SolrIndexConfig.<init> IndexWriter infoStream solr logging is enabled
   [junit4]   2> 300007 T866 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 300068 T866 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 300069 T866 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 300076 T866 oass.IndexSchema.readSchema [collection1] Schema name=example
   [junit4]   2> 300392 T866 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 300393 T866 oasr.RestManager$Registry.registerManagedResource Registered ManagedResource impl org.apache.solr.rest.schema.analysis.ManagedWordSetResource for path /schema/analysis/stopwords/english
   [junit4]   2> 300394 T866 oasr.RestManager$Registry.registerManagedResource Registered ManagedResource impl org.apache.solr.rest.schema.analysis.ManagedSynonymFilterFactory$SynonymManager for path /schema/analysis/synonyms/english
   [junit4]   2> 300406 T866 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 300409 T866 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 300438 T866 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 300438 T866 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 300438 T866 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data/
   [junit4]   2> 300438 T866 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1195d5
   [junit4]   2> 300439 T866 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for newSearcher: org.apache.solr.core.QuerySenderListener{queries=[]}
   [junit4]   2> 300439 T866 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for firstSearcher: org.apache.solr.core.QuerySenderListener{queries=[{q=static firstSearcher warming in solrconfig.xml}]}
   [junit4]   2> 300440 T866 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@dbdf2b
   [junit4]   2> 300440 T866 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 300440 T866 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 300440 T866 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 300440 T866 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 0 directories currently being tracked
   [junit4]   2> 300441 T866 oasc.CoreContainer.create ERROR Error creating core [collection1]: access denied ("java.io.FilePermission" "/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data" "write") org.apache.solr.common.SolrException: access denied ("java.io.FilePermission" "/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data" "write")
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:890)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data" "write")
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.NativeFSLockFactory.<init>(NativeFSLockFactory.java:93)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:502)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
   [junit4]   2> 	... 8 more
   [junit4]   2> 
   [junit4]   2> 300442 T824 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=29574453
   [junit4]   2> 300443 T824 oasc.Overseer.close Overseer (id=92450710388408322-188.138.97.18:8983_solr-n_0000000000) closing
   [junit4]   2> 300444 T863 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 188.138.97.18:8983_solr
   [junit4]   2> 300448 T869 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:347)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:344)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:344)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:346)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:296)
   [junit4]   2> 
   [junit4]   2> 301948 T867 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 301949 T824 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=29574453
   [junit4]   2> 301950 T824 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:41739 41739
   [junit4]   2> 302767 T852 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:41739 41739
   [junit4]   2> 302770 T824 oas.SolrTestCaseJ4.tearDown ###Ending testInitPathExists
   [junit4]   2> 302779 T824 oas.SolrTestCaseJ4.setUp ###Starting testChrootBootstrap
   [junit4]   2> 302780 T824 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 302781 T870 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 302881 T824 oasc.ZkTestServer.run start zk server on port:37449
   [junit4]   2> 302882 T824 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/'
   [junit4]   2> 302899 T824 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/solr.xml
   [junit4]   2> 302920 T824 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lu

[...truncated too long message...]

ing termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=en_CA, timezone=Europe/Gibraltar
   [junit4]   2> NOTE: Linux 3.13.0-35-generic i386/Oracle Corporation 1.7.0_67 (32-bit)/cpus=8,threads=1,free=171454224,total=281018368
   [junit4]   2> NOTE: All tests run in this JVM: [DirectSolrSpellCheckerTest, TestSuggestSpellingConverter, TestStressVersions, DirectUpdateHandlerTest, BlockCacheTest, TestReplicationHandler, UnloadDistributedZkTest, DocValuesMultiTest, TestStressRecovery, OverseerStatusTest, ParsingFieldUpdateProcessorsTest, HighlighterConfigTest, FileUtilsTest, SolrCoreCheckLockOnStartupTest, SharedFSAutoReplicaFailoverUtilsTest, TestSurroundQueryParser, HdfsRecoveryZkTest, TestCloudSchemaless, SuggesterWFSTTest, TestRecoveryHdfs, TestManagedStopFilterFactory, TestWordDelimiterFilterFactory, ShardRoutingTest, MBeansHandlerTest, DocValuesMissingTest, TestFaceting, RecoveryZkTest, ClusterStateUpdateTest, FieldFacetTest, ConnectionManagerTest, TestRequestStatusCollectionAPI, DistributedFacetPivotSmallTest, TestCollationFieldDocValues, TestDocumentBuilder, TestArbitraryIndexDir, TestFastWriter, TestSimpleQParserPlugin, HdfsSyncSliceTest, TestManagedResource, DisMaxRequestHandlerTest, OverseerRolesTest, SyncSliceTest, TestSearchPerf, NoCacheHeaderTest, MinimalSchemaTest, AbstractAnalyticsFacetTest, CoreAdminRequestStatusTest, TestSolrDeletionPolicy2, ModifyConfFileTest, TestFuzzyAnalyzedSuggestions, EchoParamsTest, TestPartialUpdateDeduplication, ShowFileRequestHandlerTest, DistributedFacetPivotLargeTest, HdfsBasicDistributedZkTest, SolrIndexSplitterTest, ExternalFileFieldSortTest, TestFieldResource, JSONWriterTest, PluginInfoTest, TestCloudManagedSchema, SolrXmlInZkTest, SSLMigrationTest, PeerSyncTest, TestRawResponseWriter, TestFastLRUCache, TestCoreDiscovery, TestSolrQueryParserResource, SignatureUpdateProcessorFactoryTest, ZkControllerTest, DeleteLastCustomShardedReplicaTest]
   [junit4] Completed on J0 in 129.29s, 1 test, 1 error <<< FAILURES!

[...truncated 463 lines...]
   [junit4] Suite: org.apache.solr.core.RAMDirectoryFactoryTest
   [junit4]   2> 1921622 T5833 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@6846fd lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@11c9a2e assuming 'simple'
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=RAMDirectoryFactoryTest -Dtests.method=test -Dtests.seed=7274912D883006C4 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ro -Dtests.timezone=Asia/Riyadh87 -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.01s J1 | RAMDirectoryFactoryTest.test <<<
   [junit4]    > Throwable #1: java.security.AccessControlException: access denied ("java.io.FilePermission" "/fake/path" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7274912D883006C4:FA20AEF726CC6B3C]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.RAMDirectoryFactoryTest.dotestOpenReturnsTheSameForSamePath(RAMDirectoryFactoryTest.java:46)
   [junit4]    > 	at org.apache.solr.core.RAMDirectoryFactoryTest.test(RAMDirectoryFactoryTest.java:33)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=ro, timezone=Asia/Riyadh87
   [junit4]   2> NOTE: Linux 3.13.0-35-generic i386/Oracle Corporation 1.7.0_67 (32-bit)/cpus=8,threads=1,free=184238488,total=434634752
   [junit4]   2> NOTE: All tests run in this JVM: [TestDistributedGrouping, MultiThreadedOCPTest, TestSolrXmlPersistor, TestCSVLoader, AsyncMigrateRouteKeyTest, QueryElevationComponentTest, TestCharFilters, VMParamsZkACLAndCredentialsProvidersTest, TestMultiCoreConfBootstrap, HdfsLockFactoryTest, UniqFieldsUpdateProcessorFactoryTest, TestCSVResponseWriter, TestSolrXml, LoggingHandlerTest, SolrIndexConfigTest, TestTolerantSearch, UUIDFieldTest, DistanceFunctionTest, TestStandardQParsers, TestRestManager, DebugComponentTest, TestZkChroot, TestReversedWildcardFilterFactory, TestAddFieldRealTimeGet, DateRangeFieldTest, URLClassifyProcessorTest, PreAnalyzedFieldTest, TestCollapseQParserPlugin, TestWriterPerf, CurrencyFieldXmlFileTest, PrimUtilsTest, AliasIntegrationTest, TestFiltering, TestNonNRTOpen, TestDynamicFieldResource, TestRecovery, SOLR749Test, TestRandomMergePolicy, QueryParsingTest, TestMergePolicyConfig, BJQParserTest, StressHdfsTest, TestPseudoReturnFields, TestPivotHelperCode, BlockDirectoryTest, CachingDirectoryFactoryTest, TestExpandComponent, RankQueryTest, SearchHandlerTest, AssignTest, UpdateRequestProcessorFactoryTest, BadCopyFieldTest, HttpPartitionTest, TestRandomFaceting, AnalyticsMergeStrategyTest, TestMiniSolrCloudCluster, SchemaVersionSpecificBehaviorTest, TestPHPSerializedResponseWriter, ExpressionTest, TestIndexingPerformance, TestOmitPositions, SimplePostToolTest, CSVRequestHandlerTest, TestJoin, TestInfoStreamLogging, TestBM25SimilarityFactory, TestPerFieldSimilarity, SolrInfoMBeanTest, ScriptEngineTest, ClusterStateTest, TestUtils, SynonymTokenizerTest, DistributedSuggestComponentTest, ReplicationFactorTest, FieldAnalysisRequestHandlerTest, DefaultValueUpdateProcessorTest, LukeRequestHandlerTest, FullSolrCloudDistribCmdsTest, TestFunctionQuery, TestModifyConfFiles, TestAtomicUpdateErrorCases, RangeFacetTest, TestReplicationHandlerBackup, BinaryUpdateRequestHandlerTest, StatsComponentTest, QueryFacetTest, SolrCoreTest, TestElisionMultitermQuery, SliceStateTest, TestIBSimilarityFactory, TestNonDefinedSimilarityFactory, ShardSplitTest, TestSerializedLuceneMatchVersion, SoftAutoCommitTest, TestManagedSynonymFilterFactory, CoreContainerCoreInitFailuresTest, HardAutoCommitTest, TestPostingsSolrHighlighter, LegacyHTMLStripCharFilterTest, SolrPluginUtilsTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, CollectionsAPIDistributedZkTest, OpenCloseCoreStressTest, OverseerTest, BasicZkTest, ZkSolrClientTest, TestRandomDVFaceting, AutoCommitTest, TestHashPartitioner, DistributedSpellCheckComponentTest, TestRealTimeGet, TestReload, TestRangeQuery, TestCoreContainer, SimpleFacetsTest, TestSolr4Spatial, SpellCheckComponentTest, TestGroupingSearch, SolrCmdDistributorTest, AnalysisAfterCoreReloadTest, TestExtendedDismaxParser, SuggesterFSTTest, CoreAdminHandlerTest, SolrRequestParserTest, DocValuesTest, SuggesterTSTTest, TestStressLucene, TestTrie, SpatialFilterTest, WordBreakSolrSpellCheckerTest, QueryEqualityTest, SortByFunctionTest, CacheHeaderTest, RequestHandlersTest, DocumentBuilderTest, PathHierarchyTokenizerFactoryTest, RequiredFieldsTest, IndexSchemaRuntimeFieldTest, TestJmxIntegration, TestAnalyzedSuggestions, PingRequestHandlerTest, TestBinaryResponseWriter, UpdateParamsTest, TestSolrIndexConfig, CopyFieldTest, MultiTermTest, SampleTest, TestBinaryField, DirectSolrConnectionTest, NotRequiredUniqueKeyTest, TestPhraseSuggestions, SpellPossibilityIteratorTest, TestCodecSupport, TestXIncludeConfig, TestSweetSpotSimilarityFactory, TestDFRSimilarityFactory, TestLMJelinekMercerSimilarityFactory, ResourceLoaderTest, OpenExchangeRatesOrgProviderTest, DateMathParserTest, TestSystemIdResolver, DateFieldTest, DOMUtilTest, RAMDirectoryFactoryTest]
   [junit4] Completed on J1 in 0.02s, 1 test, 1 error <<< FAILURES!

[...truncated 349 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:471: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:60: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1358: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:961: There were test failures: 438 suites, 1830 tests, 3 suite-level errors, 3 errors, 45 ignored (23 assumptions)

Total time: 93 minutes 25 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.7.0_67 -server -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_20) - Build # 11242 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11242/
Java: 64bit/jdk1.8.0_20 -XX:-UseCompressedOops -XX:+UseParallelGC

5 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestZkChroot

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=5391, name=coreLoadExecutor-2759-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=5391, name=coreLoadExecutor-2759-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
	at java.lang.Thread.getStackTrace(Thread.java:1552)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@6ef4213a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@39743af1)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:699)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:696)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:712)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	... 1 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestZkChroot

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=5420, name=coreLoadExecutor-2769-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=5420, name=coreLoadExecutor-2769-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
	at java.lang.Thread.getStackTrace(Thread.java:1552)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@2b785af5 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@49d59f43)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:699)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:696)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:712)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	... 1 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestZkChroot

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=5438, name=coreLoadExecutor-2776-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=5438, name=coreLoadExecutor-2776-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
	at java.lang.Thread.getStackTrace(Thread.java:1552)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@5feb2b20 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3a105ed0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:699)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:696)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:712)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	... 1 more


FAILED:  org.apache.solr.core.CachingDirectoryFactoryTest.stressTest

Error Message:
Captured an uncaught exception in thread: Thread[id=564, name=Thread-199, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=564, name=Thread-199, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path14/6" "write")
	at __randomizedtesting.SeedInfo.seed([A251D9EC507FEBB2]:0)
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
	at java.security.AccessController.checkPermission(AccessController.java:884)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
	at java.nio.file.Files.createDirectory(Files.java:674)
	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
	at java.nio.file.Files.createDirectories(Files.java:727)
	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)


FAILED:  org.apache.solr.core.RAMDirectoryFactoryTest.test

Error Message:
access denied ("java.io.FilePermission" "/fake/path" "write")

Stack Trace:
java.security.AccessControlException: access denied ("java.io.FilePermission" "/fake/path" "write")
	at __randomizedtesting.SeedInfo.seed([A251D9EC507FEBB2:2A05E636FE83864A]:0)
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
	at java.security.AccessController.checkPermission(AccessController.java:884)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
	at java.nio.file.Files.createDirectory(Files.java:674)
	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
	at java.nio.file.Files.createDirectories(Files.java:727)
	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
	at org.apache.solr.core.RAMDirectoryFactoryTest.dotestOpenReturnsTheSameForSamePath(RAMDirectoryFactoryTest.java:46)
	at org.apache.solr.core.RAMDirectoryFactoryTest.test(RAMDirectoryFactoryTest.java:33)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 10847 lines...]
   [junit4] Suite: org.apache.solr.core.CachingDirectoryFactoryTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.core.CachingDirectoryFactoryTest-A251D9EC507FEBB2-001/init-core-data-001
   [junit4]   2> 344713 T563 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 344715 T563 oas.SolrTestCaseJ4.setUp ###Starting stressTest
   [junit4]   2> 344909 T564 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@2248c5c3 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6375cff8 assuming 'simple'
   [junit4]   2> 344911 T565 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@4cacdf8b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@d91d5b0 assuming 'simple'
   [junit4]   2> 199 T564 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-199,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path14/6" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([A251D9EC507FEBB2]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 199 T565 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-200,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path14/6" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([A251D9EC507FEBB2]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 344932 T566 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@7456b2d5 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@356bb8ee assuming 'simple'
   [junit4]   2> 219 T566 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-201,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path14/6" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([A251D9EC507FEBB2]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 344933 T568 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@392ccb1e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7811c125 assuming 'simple'
   [junit4]   2> 344934 T573 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@15978e5d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@54aa8466 assuming 'simple'
   [junit4]   2> 221 T568 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-203,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path14/6" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([A251D9EC507FEBB2]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 344934 T574 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@561f69ec lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@172263d7 assuming 'simple'
   [junit4]   2> 344935 T569 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@492b139c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@81619a7 assuming 'simple'
   [junit4]   2> 221 T573 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-208,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path14/6" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([A251D9EC507FEBB2]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 344935 T571 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@4d394dff lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@c0447c4 assuming 'simple'
   [junit4]   2> 222 T574 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-209,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path14/6" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([A251D9EC507FEBB2]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 344935 T570 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@2ede7006 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6fe37a3d assuming 'simple'
   [junit4]   2> 222 T569 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-204,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path14/6" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([A251D9EC507FEBB2]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 344936 T572 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@5ead62b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@44d7dc10 assuming 'simple'
   [junit4]   2> 344936 T567 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@573dbef3 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1600b4c8 assuming 'simple'
   [junit4]   2> 223 T572 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-207,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path14/6" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([A251D9EC507FEBB2]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 223 T570 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-205,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path14/6" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([A251D9EC507FEBB2]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 223 T567 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-202,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path14/6" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([A251D9EC507FEBB2]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 227 T571 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-206,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path14/6" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([A251D9EC507FEBB2]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 352740 T581 oasc.CachingDirectoryFactory.close Closing RAMDirectoryFactory - 0 directories currently being tracked
   [junit4]   2> 352817 T563 oas.SolrTestCaseJ4.tearDown ###Ending stressTest
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CachingDirectoryFactoryTest -Dtests.method=stressTest -Dtests.seed=A251D9EC507FEBB2 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=it -Dtests.timezone=Pacific/Auckland -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   8.14s J0 | CachingDirectoryFactoryTest.stressTest <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=564, name=Thread-199, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path14/6" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A251D9EC507FEBB2]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=565, name=Thread-200, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path14/6" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A251D9EC507FEBB2]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=566, name=Thread-201, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path14/6" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A251D9EC507FEBB2]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #4: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=568, name=Thread-203, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path14/6" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A251D9EC507FEBB2]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #5: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=573, name=Thread-208, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path14/6" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A251D9EC507FEBB2]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #6: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=574, name=Thread-209, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path14/6" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A251D9EC507FEBB2]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #7: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=569, name=Thread-204, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path14/6" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A251D9EC507FEBB2]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #8: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=570, name=Thread-205, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path14/6" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A251D9EC507FEBB2]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #9: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=572, name=Thread-207, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path14/6" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A251D9EC507FEBB2]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #10: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=567, name=Thread-202, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path14/6" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A251D9EC507FEBB2]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #11: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=571, name=Thread-206, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path14/6" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A251D9EC507FEBB2]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 352857 T563 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=it, timezone=Pacific/Auckland
   [junit4]   2> NOTE: Linux 3.13.0-35-generic amd64/Oracle Corporation 1.8.0_20 (64-bit)/cpus=8,threads=1,free=182818624,total=324534272
   [junit4]   2> NOTE: All tests run in this JVM: [HdfsBasicDistributedZk2Test, TestSolrQueryParserResource, TestStressUserVersions, DocValuesMultiTest, PeerSyncTest, TestSort, TestConfig, TestHighFrequencyDictionaryFactory, TestTrie, HdfsBasicDistributedZkTest, AbstractAnalyticsStatsTest, TestIntervalFaceting, SuggesterTest, FileUtilsTest, HdfsSyncSliceTest, HighlighterConfigTest, ZkControllerTest, TestShardHandlerFactory, DistribDocExpirationUpdateProcessorTest, RecoveryZkTest, DistributedDebugComponentTest, CachingDirectoryFactoryTest]
   [junit4] Completed on J0 in 8.17s, 1 test, 1 error <<< FAILURES!

[...truncated 556 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestZkChroot
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestZkChroot-A251D9EC507FEBB2-001/init-core-data-001
   [junit4]   2> 1833273 T5378 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
   [junit4]   2> 1833276 T5378 oas.SolrTestCaseJ4.setUp ###Starting testChrootBootstrap
   [junit4]   2> 1833277 T5378 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1833277 T5379 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1833377 T5378 oasc.ZkTestServer.run start zk server on port:40200
   [junit4]   2> 1833378 T5378 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/'
   [junit4]   2> 1833389 T5378 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/solr.xml
   [junit4]   2> 1833393 T5378 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr
   [junit4]   2> 1833394 T5378 oasc.CoreContainer.<init> New CoreContainer 1150765859
   [junit4]   2> 1833394 T5378 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/]
   [junit4]   2> 1833394 T5378 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 1833394 T5378 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 1833394 T5378 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 1833395 T5378 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1833395 T5378 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1833395 T5378 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1833395 T5378 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1833395 T5378 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1833395 T5378 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1833396 T5378 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=0&connTimeout=0&retry=false
   [junit4]   2> 1833396 T5378 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1833397 T5378 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1833397 T5378 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 1833398 T5378 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40200/foo/bar
   [junit4]   2> 1833398 T5378 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1833398 T5378 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1833399 T5378 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1833400 T5385 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@53bceade name:ZooKeeperConnection Watcher:127.0.0.1:40200 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1833401 T5378 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1833402 T5378 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1833402 T5378 oascc.SolrZkClient.makePath makePath: /foo/bar
   [junit4]   2> 1833406 T5378 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1833407 T5387 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@63866eca name:ZooKeeperConnection Watcher:127.0.0.1:40200/foo/bar got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1833407 T5378 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1833408 T5378 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1833409 T5378 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1833410 T5378 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1833411 T5378 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1833413 T5378 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1833414 T5378 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1833415 T5378 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/188.138.97.18:8983_solr
   [junit4]   2> 1833415 T5378 oascc.SolrZkClient.makePath makePath: /live_nodes/188.138.97.18:8983_solr
   [junit4]   2> 1833416 T5378 oascc.SolrZkClient.makePath makePath: /collections
   [junit4]   2> 1833417 T5378 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1833418 T5378 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1833419 T5378 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1833420 T5378 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 188.138.97.18:8983_solr
   [junit4]   2> 1833420 T5378 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1833421 T5378 oasc.Overseer.start Overseer (id=92450016226639873-188.138.97.18:8983_solr-n_0000000000) starting
   [junit4]   2> 1833422 T5378 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1833425 T5378 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1833426 T5389 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1833426 T5378 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1833427 T5378 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1833427 T5378 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1833428 T5378 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr
   [junit4]   2> 1833429 T5388 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1833433 T5378 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/
   [junit4]   2> 1833433 T5378 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1833433 T5378 oasc.ZkController.bootstrapConf bootstrapping config for 1 cores into ZooKeeper using solr.xml from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/
   [junit4]   2> 1833434 T5378 oasc.ZkController.bootstrapConf Uploading directory /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/conf with name collection1 for SolrCore collection1
   [junit4]   2> 1833434 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/spellings.txt
   [junit4]   2> 1833437 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/stopwords.txt
   [junit4]   2> 1833438 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/scripts.conf
   [junit4]   2> 1833440 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/admin-extra.html
   [junit4]   2> 1833441 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/schema.xml
   [junit4]   2> 1833444 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/solrconfig.xml
   [junit4]   2> 1833446 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/admin-extra.menu-top.html
   [junit4]   2> 1833448 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/elevate.xml
   [junit4]   2> 1833450 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/_schema_analysis_synonyms_english.json
   [junit4]   2> 1833451 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_nl.txt
   [junit4]   2> 1833454 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_eu.txt
   [junit4]   2> 1833456 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ga.txt
   [junit4]   2> 1833457 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_fr.txt
   [junit4]   2> 1833459 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_bg.txt
   [junit4]   2> 1833462 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stemdict_nl.txt
   [junit4]   2> 1833465 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/contractions_ca.txt
   [junit4]   2> 1833468 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ja.txt
   [junit4]   2> 1833470 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_id.txt
   [junit4]   2> 1833472 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_it.txt
   [junit4]   2> 1833475 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_fi.txt
   [junit4]   2> 1833477 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/contractions_fr.txt
   [junit4]   2> 1833479 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_el.txt
   [junit4]   2> 1833481 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_en.txt
   [junit4]   2> 1833482 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_da.txt
   [junit4]   2> 1833484 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stoptags_ja.txt
   [junit4]   2> 1833485 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_no.txt
   [junit4]   2> 1833486 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_pt.txt
   [junit4]   2> 1833488 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/hyphenations_ga.txt
   [junit4]   2> 1833489 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ru.txt
   [junit4]   2> 1833490 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ro.txt
   [junit4]   2> 1833491 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_cz.txt
   [junit4]   2> 1833493 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_hu.txt
   [junit4]   2> 1833494 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_th.txt
   [junit4]   2> 1833495 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ckb.txt
   [junit4]   2> 1833497 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_gl.txt
   [junit4]   2> 1833498 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ar.txt
   [junit4]   2> 1833499 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_tr.txt
   [junit4]   2> 1833501 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_es.txt
   [junit4]   2> 1833503 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/userdict_ja.txt
   [junit4]   2> 1833505 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/contractions_it.txt
   [junit4]   2> 1833506 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ca.txt
   [junit4]   2> 1833508 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_lv.txt
   [junit4]   2> 1833510 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_de.txt
   [junit4]   2> 1833513 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/contractions_ga.txt
   [junit4]   2> 1833515 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_hi.txt
   [junit4]   2> 1833516 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_hy.txt
   [junit4]   2> 1833519 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_sv.txt
   [junit4]   2> 1833521 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_fa.txt
   [junit4]   2> 1833523 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/synonyms.txt
   [junit4]   2> 1833525 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/update-script.js
   [junit4]   2> 1833527 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facet_pivot.vm
   [junit4]   2> 1833529 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/jquery.autocomplete.css
   [junit4]   2> 1833531 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/debug.vm
   [junit4]   2> 1833533 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/query_form.vm
   [junit4]   2> 1833536 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/pagination_top.vm
   [junit4]   2> 1833539 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/did_you_mean.vm
   [junit4]   2> 1833541 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/query_spatial.vm
   [junit4]   2> 1833545 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/hit_plain.vm
   [junit4]   2> 1833548 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/pagination_bottom.vm
   [junit4]   2> 1833551 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/header.vm
   [junit4]   2> 1833554 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/layout.vm
   [junit4]   2> 1833556 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/suggest.vm
   [junit4]   2> 1833558 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/README.txt
   [junit4]   2> 1833560 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/head.vm
   [junit4]   2> 1833562 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/footer.vm
   [junit4]   2> 1833565 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/results_list.vm
   [junit4]   2> 1833568 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/jquery.autocomplete.js
   [junit4]   2> 1833571 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/hit_grouped.vm
   [junit4]   2> 1833575 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facet_fields.vm
   [junit4]   2> 1833578 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/query_group.vm
   [junit4]   2> 1833582 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/query.vm
   [junit4]   2> 1833585 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/cluster_results.vm
   [junit4]   2> 1833587 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/VM_global_library.vm
   [junit4]   2> 1833589 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facet_queries.vm
   [junit4]   2> 1833590 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/mime_type_lists.vm
   [junit4]   2> 1833592 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/error.vm
   [junit4]   2> 1833594 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/product_doc.vm
   [junit4]   2> 1833596 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/main.css
   [junit4]   2> 1833598 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/browse.vm
   [junit4]   2> 1833600 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/richtext_doc.vm
   [junit4]   2> 1833602 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/cluster.vm
   [junit4]   2> 1833604 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facets.vm
   [junit4]   2> 1833607 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/tabs.vm
   [junit4]   2> 1833609 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/join_doc.vm
   [junit4]   2> 1833611 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/hit.vm
   [junit4]   2> 1833613 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facet_ranges.vm
   [junit4]   2> 1833616 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1833618 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/_schema_analysis_stopwords_english.json
   [junit4]   2> 1833621 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/protwords.txt
   [junit4]   2> 1833624 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/_rest_managed.json
   [junit4]   2> 1833627 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/example.xsl
   [junit4]   2> 1833630 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/example_rss.xsl
   [junit4]   2> 1833632 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/updateXml.xsl
   [junit4]   2> 1833634 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/luke.xsl
   [junit4]   2> 1833636 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/example_atom.xsl
   [junit4]   2> 1833639 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/mapping-FoldToASCII.txt
   [junit4]   2> 1833643 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/clustering/carrot2/stc-attributes.xml
   [junit4]   2> 1833646 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/clustering/carrot2/lingo-attributes.xml
   [junit4]   2> 1833649 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/clustering/carrot2/kmeans-attributes.xml
   [junit4]   2> 1833654 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/admin-extra.menu-bottom.html
   [junit4]   2> 1833656 T5378 oascc.SolrZkClient.makePath makePath: /configs/collection1/currency.xml
   [junit4]   2> 1833660 T5378 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr
   [junit4]   2> 1833663 T5378 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/
   [junit4]   2> 1833664 T5378 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1833665 T5391 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1833665 T5391 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1833666 T5391 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1833666 T5392 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1833669 T5388 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://188.138.97.18:8983/solr",
   [junit4]   2> 	  "node_name":"188.138.97.18:8983_solr",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1833669 T5388 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1833672 T5392 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1834667 T5391 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1834667 T5391 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1834667 T5391 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:collection1
   [junit4]   2> 1834668 T5391 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1834670 T5391 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1834671 T5391 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=collection1 specified config exists in ZooKeeper
   [junit4]   2> 1834671 T5391 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/'
   [junit4]   2> 1834687 T5391 oasc.SolrConfig.initLibs Adding specified lib dirs to ClassLoader
   [junit4]   2> 1834688 T5391 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/extraction/lib (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/extraction/lib).
   [junit4]   2> 1834689 T5391 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 1834690 T5391 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/clustering/lib/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/clustering/lib).
   [junit4]   2> 1834690 T5391 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 1834690 T5391 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/langid/lib/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/langid/lib).
   [junit4]   2> 1834690 T5391 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 1834691 T5391 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/velocity/lib (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/velocity/lib).
   [junit4]   2> 1834691 T5391 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 1834716 T5391 oasu.SolrIndexConfig.<init> IndexWriter infoStream solr logging is enabled
   [junit4]   2> 1834719 T5391 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1834761 T5391 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1834763 T5391 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1834767 T5391 oass.IndexSchema.readSchema [collection1] Schema name=example
   [junit4]   2> 1835436 T5391 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1835437 T5391 oasr.RestManager$Registry.registerManagedResource Registered ManagedResource impl org.apache.solr.rest.schema.analysis.ManagedWordSetResource for path /schema/analysis/stopwords/english
   [junit4]   2> 1835438 T5391 oasr.RestManager$Registry.registerManagedResource Registered ManagedResource impl org.apache.solr.rest.schema.analysis.ManagedSynonymFilterFactory$SynonymManager for path /schema/analysis/synonyms/english
   [junit4]   2> 1835452 T5391 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1835454 T5391 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1835499 T5391 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1835499 T5391 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1835500 T5391 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data/
   [junit4]   2> 1835500 T5391 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4d8c6632
   [junit4]   2> 1835501 T5391 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for newSearcher: org.apache.solr.core.QuerySenderListener{queries=[]}
   [junit4]   2> 1835501 T5391 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for firstSearcher: org.apache.solr.core.QuerySenderListener{queries=[{q=static firstSearcher warming in solrconfig.xml}]}
   [junit4]   2> 1835502 T5391 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@324c1c87
   [junit4]   2> 1835503 T5391 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 1835503 T5391 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1835504 T5391 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1835504 T5391 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 0 directories currently being tracked
   [junit4]   2> 1835505 T5391 oasc.CoreContainer.create ERROR Error creating core [collection1]: access denied ("java.io.FilePermission" "/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data" "write") org.apache.solr.common.SolrException: access denied ("java.io.FilePermission" "/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data" "write")
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:891)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data" "write")
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.NativeFSLockFactory.<init>(NativeFSLockFactory.java:93)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:502)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
   [junit4]   2> 	... 8 more
   [junit4]   2> 
   [junit4]   2> 1835509 T5378 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1835509 T5378 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1835511 T5395 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@cb5894 name:ZooKeeperConnection Watcher:127.0.0.1:40200 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1835511 T5378 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1835512 T5378 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1835513 T5378 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1150765859
   [junit4]   2> 1835513 T5378 oasc.Overseer.close Overseer (id=92450016226639873-188.138.97.18:8983_solr-n_0000000000) closing
   [junit4]   2> 1835514 T5388 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 188.138.97.18:8983_solr
   [junit4]   2> 1837015 T5396 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:347)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:344)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:344)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:346)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:296)
   [junit4]   2> 
   [junit4]   2> 1837015 T5392 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1837018 T5378 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1150765859
   [junit4]   2> 1837019 T5378 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:40200 40200
   [junit4]   2> 1837686 T5379 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:40200 40200
   [junit4]   2> 1837688 T5378 oas.SolrTestCaseJ4.tearDown ###Ending testChrootBootstrap
   [junit4]   2> 1837692 T5378 oas.SolrTes

[...truncated too long message...]

Thread.getStackTrace(Thread.java:1552)
   [junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
   [junit4]    > 	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
   [junit4]    > 	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
   [junit4]    > 	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
   [junit4]    > 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
   [junit4]    > 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
   [junit4]    > 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
   [junit4]    > 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@5feb2b20 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3a105ed0)
   [junit4]    > 	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
   [junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
   [junit4]    > 	... 1 more
   [junit4] Completed on J0 in 11.87s, 4 tests, 3 errors <<< FAILURES!

[...truncated 317 lines...]
   [junit4] Suite: org.apache.solr.core.RAMDirectoryFactoryTest
   [junit4]   2> 2134796 T6306 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@617dc7fb lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@61f488b4 assuming 'simple'
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=RAMDirectoryFactoryTest -Dtests.method=test -Dtests.seed=A251D9EC507FEBB2 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=mk_MK -Dtests.timezone=WET -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.02s J0 | RAMDirectoryFactoryTest.test <<<
   [junit4]    > Throwable #1: java.security.AccessControlException: access denied ("java.io.FilePermission" "/fake/path" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A251D9EC507FEBB2:2A05E636FE83864A]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.RAMDirectoryFactoryTest.dotestOpenReturnsTheSameForSamePath(RAMDirectoryFactoryTest.java:46)
   [junit4]    > 	at org.apache.solr.core.RAMDirectoryFactoryTest.test(RAMDirectoryFactoryTest.java:33)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=mk_MK, timezone=WET
   [junit4]   2> NOTE: Linux 3.13.0-35-generic amd64/Oracle Corporation 1.8.0_20 (64-bit)/cpus=8,threads=1,free=45562832,total=408420352
   [junit4]   2> NOTE: All tests run in this JVM: [HdfsBasicDistributedZk2Test, TestSolrQueryParserResource, TestStressUserVersions, DocValuesMultiTest, PeerSyncTest, TestSort, TestConfig, TestHighFrequencyDictionaryFactory, TestTrie, HdfsBasicDistributedZkTest, AbstractAnalyticsStatsTest, TestIntervalFaceting, SuggesterTest, FileUtilsTest, HdfsSyncSliceTest, HighlighterConfigTest, ZkControllerTest, TestShardHandlerFactory, DistribDocExpirationUpdateProcessorTest, RecoveryZkTest, DistributedDebugComponentTest, CachingDirectoryFactoryTest, PathHierarchyTokenizerFactoryTest, TestAnalyzedSuggestions, TestSolr4Spatial, TestSolrCoreProperties, PluginInfoTest, RemoteQueryErrorTest, ShowFileRequestHandlerTest, TestLRUCache, RollingRestartTest, TestFastOutputStream, HighlighterTest, TestRandomDVFaceting, DOMUtilTest, SpellCheckComponentTest, ExternalFileFieldSortTest, SolrCoreCheckLockOnStartupTest, TestFreeTextSuggestions, TestFastLRUCache, BasicDistributedZkTest, ChangedSchemaMergeTest, ResponseHeaderTest, TestDefaultSimilarityFactory, TestSolrXmlPersistence, BadIndexSchemaTest, TestQuerySenderListener, NotRequiredUniqueKeyTest, TestManagedSynonymFilterFactory, ExpressionTest, SuggesterTSTTest, TestJmxIntegration, TestManagedSchemaFieldTypeResource, CoreAdminHandlerTest, BlockCacheTest, ShardRoutingTest, TestReloadAndDeleteDocs, SignatureUpdateProcessorFactoryTest, MoreLikeThisHandlerTest, TestSchemaResource, MinimalSchemaTest, TestWordDelimiterFilterFactory, SuggesterWFSTTest, DocumentBuilderTest, XmlUpdateRequestHandlerTest, TestCoreDiscovery, ExternalCollectionsTest, CursorMarkTest, RegexBoostProcessorTest, FieldMutatingUpdateProcessorTest, XsltUpdateRequestHandlerTest, QueryResultKeyTest, OutOfBoxZkACLAndCredentialsProvidersTest, HdfsUnloadDistributedZkTest, TestDynamicFieldResource, TestAnalyzeInfixSuggestions, HdfsDirectoryTest, DistributedQueryComponentCustomSortTest, DirectUpdateHandlerTest, NoCacheHeaderTest, BadComponentTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestBinaryResponseWriter, OpenExchangeRatesOrgProviderTest, TestSchemaSimilarityResource, DisMaxRequestHandlerTest, TestMiniSolrCloudCluster, SpellPossibilityIteratorTest, FastVectorHighlighterTest, TestMissingGroups, TimeZoneUtilsTest, ReturnFieldsTest, SpellingQueryConverterTest, DeleteLastCustomShardedReplicaTest, TestFoldingMultitermQuery, SolrIndexSplitterTest, LeaderElectionTest, TestComponentsName, DeleteShardTest, TestShortCircuitedRequests, StatelessScriptUpdateProcessorFactoryTest, DistributedFacetPivotLargeTest, ZkNodePropsTest, PrimitiveFieldTypeTest, SimpleFacetsTest, DistributedExpandComponentTest, TestStressRecovery, TestSuggestSpellingConverter, TestReload, ChaosMonkeySafeLeaderTest, UnloadDistributedZkTest, SyncSliceTest, OpenCloseCoreStressTest, OverseerTest, TestReplicationHandler, ClusterStateUpdateTest, ZkSolrClientTest, TestZkChroot, ZkCLITest, TestDistributedSearch, TestRecovery, DistributedSpellCheckComponentTest, TestRealTimeGet, TestJoin, SolrCoreTest, TestGroupingSearch, ConvertedLegacyTest, TestFiltering, OverseerCollectionProcessorTest, CurrencyFieldOpenExchangeTest, DistributedQueryElevationComponentTest, TestSolrDeletionPolicy1, SolrInfoMBeanTest, CacheHeaderTest, DirectSolrSpellCheckerTest, FileBasedSpellCheckerTest, RequestHandlersTest, TestSolrQueryParser, IndexSchemaRuntimeFieldTest, IndexSchemaTest, JSONWriterTest, MBeansHandlerTest, JsonLoaderTest, TestPartialUpdateDeduplication, PingRequestHandlerTest, TestLFUCache, UpdateParamsTest, TestSolrIndexConfig, ResponseLogComponentTest, TestMergePolicyConfig, SampleTest, TestBinaryField, DateMathParserTest, TestSystemIdResolver, PrimUtilsTest, DateFieldTest, RAMDirectoryFactoryTest]
   [junit4] Completed on J0 in 0.03s, 1 test, 1 error <<< FAILURES!

[...truncated 348 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:471: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:60: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1358: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:961: There were test failures: 438 suites, 1830 tests, 3 suite-level errors, 2 errors, 45 ignored (23 assumptions)

Total time: 88 minutes 18 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_20 -XX:-UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_67) - Build # 11241 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11241/
Java: 64bit/jdk1.7.0_67 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

5 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestZkChroot

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=4566, name=coreLoadExecutor-2685-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=4566, name=coreLoadExecutor-2685-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
	at java.lang.Thread.getStackTrace(Thread.java:1589)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@7241e48d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e0eca1b)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:699)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:696)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:712)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	... 1 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestZkChroot

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=4586, name=coreLoadExecutor-2692-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=4586, name=coreLoadExecutor-2692-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
	at java.lang.Thread.getStackTrace(Thread.java:1589)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@4bb6bd98 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4e6096b8)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:699)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:696)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:712)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	... 1 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestZkChroot

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=4604, name=coreLoadExecutor-2699-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=4604, name=coreLoadExecutor-2699-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
	at java.lang.Thread.getStackTrace(Thread.java:1589)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@4ad2543e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2747323b)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:699)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:696)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:712)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	... 1 more


FAILED:  org.apache.solr.core.CachingDirectoryFactoryTest.stressTest

Error Message:
Captured an uncaught exception in thread: Thread[id=7034, name=Thread-2900, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=7034, name=Thread-2900, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1" "write")
	at __randomizedtesting.SeedInfo.seed([D93C69EAE6E4DD07]:0)
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
	at java.security.AccessController.checkPermission(AccessController.java:559)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
	at java.nio.file.Files.createDirectory(Files.java:630)
	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
	at java.nio.file.Files.createDirectories(Files.java:683)
	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)


FAILED:  org.apache.solr.core.RAMDirectoryFactoryTest.test

Error Message:
access denied ("java.io.FilePermission" "/fake/path" "write")

Stack Trace:
java.security.AccessControlException: access denied ("java.io.FilePermission" "/fake/path" "write")
	at __randomizedtesting.SeedInfo.seed([D93C69EAE6E4DD07:516856304818B0FF]:0)
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
	at java.security.AccessController.checkPermission(AccessController.java:559)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
	at java.nio.file.Files.createDirectory(Files.java:630)
	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
	at java.nio.file.Files.createDirectories(Files.java:683)
	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
	at org.apache.solr.core.RAMDirectoryFactoryTest.dotestOpenReturnsTheSameForSamePath(RAMDirectoryFactoryTest.java:46)
	at org.apache.solr.core.RAMDirectoryFactoryTest.test(RAMDirectoryFactoryTest.java:33)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11241 lines...]
   [junit4] Suite: org.apache.solr.core.RAMDirectoryFactoryTest
   [junit4]   2> 1459682 T3758 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@48b57bc3 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3f7610f1 assuming 'simple'
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=RAMDirectoryFactoryTest -Dtests.method=test -Dtests.seed=D93C69EAE6E4DD07 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_BO -Dtests.timezone=Europe/Dublin -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.03s J1 | RAMDirectoryFactoryTest.test <<<
   [junit4]    > Throwable #1: java.security.AccessControlException: access denied ("java.io.FilePermission" "/fake/path" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([D93C69EAE6E4DD07:516856304818B0FF]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.RAMDirectoryFactoryTest.dotestOpenReturnsTheSameForSamePath(RAMDirectoryFactoryTest.java:46)
   [junit4]    > 	at org.apache.solr.core.RAMDirectoryFactoryTest.test(RAMDirectoryFactoryTest.java:33)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{}, sim=DefaultSimilarity, locale=es_BO, timezone=Europe/Dublin
   [junit4]   2> NOTE: Linux 3.13.0-35-generic amd64/Oracle Corporation 1.7.0_67 (64-bit)/cpus=8,threads=1,free=64528136,total=372711424
   [junit4]   2> NOTE: All tests run in this JVM: [QueryElevationComponentTest, TestUtils, TestCloudManagedSchemaConcurrent, DeleteReplicaTest, TestShardHandlerFactory, TestImplicitCoreProperties, CollectionsAPIDistributedZkTest, TestLazyCores, ConvertedLegacyTest, SolrRequestParserTest, TestRandomMergePolicy, DistributedExpandComponentTest, AliasIntegrationTest, TestCollationField, QueryParsingTest, RequestHandlersTest, CircularListTest, TestQueryUtils, TestFieldTypeCollectionResource, TestChildDocTransformer, ShardRoutingCustomTest, PrimUtilsTest, VMParamsZkACLAndCredentialsProvidersTest, DistributedQueryComponentOptimizationTest, DistributedQueryComponentCustomSortTest, TestOmitPositions, CSVRequestHandlerTest, HdfsDirectoryTest, TestNRTOpen, DateMathParserTest, TestQuerySenderNoQuery, TermVectorComponentDistributedTest, TestBlendedInfixSuggestions, MultiThreadedOCPTest, TestFunctionQuery, HdfsUnloadDistributedZkTest, TestSolr4Spatial2, TestComplexPhraseQParserPlugin, CurrencyFieldXmlFileTest, RollingRestartTest, TestTolerantSearch, TestPivotHelperCode, IndexSchemaRuntimeFieldTest, TestStressLucene, TestSolrQueryParser, DocValuesTest, SolrIndexConfigTest, TestRestManager, TestCoreContainer, HardAutoCommitTest, FullSolrCloudDistribCmdsTest, UniqFieldsUpdateProcessorFactoryTest, DirectUpdateHandlerOptimizeTest, BJQParserTest, BlockDirectoryTest, ShardSplitTest, LegacyHTMLStripCharFilterTest, TestLFUCache, RangeFacetTest, TestManagedSchemaFieldResource, CoreContainerCoreInitFailuresTest, AnalyticsQueryTest, TestPHPSerializedResponseWriter, OpenCloseCoreStressTest, SearchHandlerTest, TriLevelCompositeIdRoutingTest, TestShortCircuitedRequests, PreAnalyzedFieldTest, OverriddenZkACLAndCredentialsProvidersTest, RAMDirectoryFactoryTest]
   [junit4] Completed on J1 in 0.05s, 1 test, 1 error <<< FAILURES!

[...truncated 101 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestZkChroot
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.TestZkChroot-D93C69EAE6E4DD07-001/init-core-data-001
   [junit4]   2> 1810351 T4544 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 1810354 T4544 oas.SolrTestCaseJ4.setUp ###Starting testNoBootstrapConf
   [junit4]   2> 1810354 T4544 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1810355 T4545 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1810455 T4544 oasc.ZkTestServer.run start zk server on port:57065
   [junit4]   2> 1810455 T4544 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1810456 T4544 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1810458 T4551 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@394b2368 name:ZooKeeperConnection Watcher:127.0.0.1:57065 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1810458 T4544 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1810458 T4544 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1810459 T4544 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/'
   [junit4]   2> 1810472 T4544 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/solr.xml
   [junit4]   2> 1810498 T4544 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr
   [junit4]   2> 1810498 T4544 oasc.CoreContainer.<init> New CoreContainer 171509989
   [junit4]   2> 1810498 T4544 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/]
   [junit4]   2> 1810499 T4544 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 1810499 T4544 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 1810499 T4544 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 1810500 T4544 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1810500 T4544 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1810500 T4544 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1810500 T4544 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1810500 T4544 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1810500 T4544 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1810502 T4544 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=0&connTimeout=0&retry=false
   [junit4]   2> 1810502 T4544 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1810503 T4544 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1810503 T4544 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 1810503 T4544 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57065/foo/bar2
   [junit4]   2> 1810503 T4544 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1810503 T4544 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1810504 T4544 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1810505 T4553 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3997c81 name:ZooKeeperConnection Watcher:127.0.0.1:57065 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1810506 T4544 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1810506 T4544 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1810508 T4544 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:57065 57065
   [junit4]   2> 1811059 T4545 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:57065 57065
   [junit4]   2> 1811060 T4544 oas.SolrTestCaseJ4.tearDown ###Ending testNoBootstrapConf
   [junit4]   2> 1811065 T4544 oas.SolrTestCaseJ4.setUp ###Starting testChrootBootstrap
   [junit4]   2> 1811065 T4544 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1811066 T4554 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1811166 T4544 oasc.ZkTestServer.run start zk server on port:42445
   [junit4]   2> 1811166 T4544 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/'
   [junit4]   2> 1811182 T4544 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/solr.xml
   [junit4]   2> 1811194 T4544 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr
   [junit4]   2> 1811195 T4544 oasc.CoreContainer.<init> New CoreContainer 907018964
   [junit4]   2> 1811195 T4544 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/]
   [junit4]   2> 1811196 T4544 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 1811196 T4544 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 1811196 T4544 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 1811196 T4544 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1811196 T4544 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1811197 T4544 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1811197 T4544 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1811197 T4544 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1811197 T4544 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1811198 T4544 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=0&connTimeout=0&retry=false
   [junit4]   2> 1811199 T4544 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1811199 T4544 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1811199 T4544 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 1811200 T4544 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42445/foo/bar
   [junit4]   2> 1811200 T4544 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1811200 T4544 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1811201 T4544 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1811203 T4560 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d887ccb name:ZooKeeperConnection Watcher:127.0.0.1:42445 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1811204 T4544 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1811204 T4544 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1811205 T4544 oascc.SolrZkClient.makePath makePath: /foo/bar
   [junit4]   2> 1811212 T4544 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1811213 T4562 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4c49f9c name:ZooKeeperConnection Watcher:127.0.0.1:42445/foo/bar got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1811213 T4544 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1811216 T4544 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1811219 T4544 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1811222 T4544 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1811225 T4544 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1811228 T4544 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1811231 T4544 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1811232 T4544 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/188.138.97.18:8983_solr
   [junit4]   2> 1811233 T4544 oascc.SolrZkClient.makePath makePath: /live_nodes/188.138.97.18:8983_solr
   [junit4]   2> 1811236 T4544 oascc.SolrZkClient.makePath makePath: /collections
   [junit4]   2> 1811239 T4544 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1811242 T4544 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1811244 T4544 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1811247 T4544 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 188.138.97.18:8983_solr
   [junit4]   2> 1811247 T4544 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1811250 T4544 oasc.Overseer.start Overseer (id=92449228915998721-188.138.97.18:8983_solr-n_0000000000) starting
   [junit4]   2> 1811253 T4544 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1811264 T4544 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1811265 T4564 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1811266 T4544 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1811268 T4544 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1811270 T4544 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1811274 T4544 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr
   [junit4]   2> 1811274 T4563 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1811279 T4544 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/
   [junit4]   2> 1811280 T4544 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1811280 T4544 oasc.ZkController.bootstrapConf bootstrapping config for 1 cores into ZooKeeper using solr.xml from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/
   [junit4]   2> 1811281 T4544 oasc.ZkController.bootstrapConf Uploading directory /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/conf with name collection1 for SolrCore collection1
   [junit4]   2> 1811282 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/spellings.txt
   [junit4]   2> 1811286 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/stopwords.txt
   [junit4]   2> 1811290 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/scripts.conf
   [junit4]   2> 1811293 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/admin-extra.html
   [junit4]   2> 1811297 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/schema.xml
   [junit4]   2> 1811300 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/solrconfig.xml
   [junit4]   2> 1811305 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/admin-extra.menu-top.html
   [junit4]   2> 1811308 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/elevate.xml
   [junit4]   2> 1811310 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/_schema_analysis_synonyms_english.json
   [junit4]   2> 1811313 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_nl.txt
   [junit4]   2> 1811317 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_eu.txt
   [junit4]   2> 1811320 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ga.txt
   [junit4]   2> 1811323 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_fr.txt
   [junit4]   2> 1811326 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_bg.txt
   [junit4]   2> 1811329 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stemdict_nl.txt
   [junit4]   2> 1811332 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/contractions_ca.txt
   [junit4]   2> 1811335 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ja.txt
   [junit4]   2> 1811338 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_id.txt
   [junit4]   2> 1811341 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_it.txt
   [junit4]   2> 1811344 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_fi.txt
   [junit4]   2> 1811348 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/contractions_fr.txt
   [junit4]   2> 1811351 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_el.txt
   [junit4]   2> 1811354 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_en.txt
   [junit4]   2> 1811356 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_da.txt
   [junit4]   2> 1811359 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stoptags_ja.txt
   [junit4]   2> 1811375 T3 oasc.CoreContainer.finalize ERROR CoreContainer was not close prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!  instance=171509989
   [junit4]   2> 1811376 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_no.txt
   [junit4]   2> 1811379 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_pt.txt
   [junit4]   2> 1811382 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/hyphenations_ga.txt
   [junit4]   2> 1811384 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ru.txt
   [junit4]   2> 1811386 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ro.txt
   [junit4]   2> 1811389 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_cz.txt
   [junit4]   2> 1811391 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_hu.txt
   [junit4]   2> 1811393 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_th.txt
   [junit4]   2> 1811394 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ckb.txt
   [junit4]   2> 1811397 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_gl.txt
   [junit4]   2> 1811399 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ar.txt
   [junit4]   2> 1811402 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_tr.txt
   [junit4]   2> 1811404 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_es.txt
   [junit4]   2> 1811407 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/userdict_ja.txt
   [junit4]   2> 1811410 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/contractions_it.txt
   [junit4]   2> 1811413 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ca.txt
   [junit4]   2> 1811416 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_lv.txt
   [junit4]   2> 1811419 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_de.txt
   [junit4]   2> 1811422 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/contractions_ga.txt
   [junit4]   2> 1811425 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_hi.txt
   [junit4]   2> 1811429 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_hy.txt
   [junit4]   2> 1811432 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_sv.txt
   [junit4]   2> 1811435 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_fa.txt
   [junit4]   2> 1811438 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/synonyms.txt
   [junit4]   2> 1811440 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/update-script.js
   [junit4]   2> 1811443 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facet_pivot.vm
   [junit4]   2> 1811447 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/jquery.autocomplete.css
   [junit4]   2> 1811450 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/debug.vm
   [junit4]   2> 1811452 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/query_form.vm
   [junit4]   2> 1811456 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/pagination_top.vm
   [junit4]   2> 1811459 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/did_you_mean.vm
   [junit4]   2> 1811462 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/query_spatial.vm
   [junit4]   2> 1811465 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/hit_plain.vm
   [junit4]   2> 1811468 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/pagination_bottom.vm
   [junit4]   2> 1811470 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/header.vm
   [junit4]   2> 1811472 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/layout.vm
   [junit4]   2> 1811475 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/suggest.vm
   [junit4]   2> 1811478 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/README.txt
   [junit4]   2> 1811481 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/head.vm
   [junit4]   2> 1811484 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/footer.vm
   [junit4]   2> 1811487 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/results_list.vm
   [junit4]   2> 1811490 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/jquery.autocomplete.js
   [junit4]   2> 1811493 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/hit_grouped.vm
   [junit4]   2> 1811496 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facet_fields.vm
   [junit4]   2> 1811499 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/query_group.vm
   [junit4]   2> 1811502 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/query.vm
   [junit4]   2> 1811505 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/cluster_results.vm
   [junit4]   2> 1811508 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/VM_global_library.vm
   [junit4]   2> 1811511 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facet_queries.vm
   [junit4]   2> 1811514 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/mime_type_lists.vm
   [junit4]   2> 1811518 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/error.vm
   [junit4]   2> 1811521 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/product_doc.vm
   [junit4]   2> 1811524 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/main.css
   [junit4]   2> 1811528 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/browse.vm
   [junit4]   2> 1811531 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/richtext_doc.vm
   [junit4]   2> 1811534 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/cluster.vm
   [junit4]   2> 1811537 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facets.vm
   [junit4]   2> 1811541 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/tabs.vm
   [junit4]   2> 1811545 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/join_doc.vm
   [junit4]   2> 1811549 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/hit.vm
   [junit4]   2> 1811552 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facet_ranges.vm
   [junit4]   2> 1811555 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1811557 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/_schema_analysis_stopwords_english.json
   [junit4]   2> 1811560 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/protwords.txt
   [junit4]   2> 1811562 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/_rest_managed.json
   [junit4]   2> 1811565 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/example.xsl
   [junit4]   2> 1811568 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/example_rss.xsl
   [junit4]   2> 1811571 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/updateXml.xsl
   [junit4]   2> 1811574 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/luke.xsl
   [junit4]   2> 1811577 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/example_atom.xsl
   [junit4]   2> 1811580 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/mapping-FoldToASCII.txt
   [junit4]   2> 1811583 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/clustering/carrot2/stc-attributes.xml
   [junit4]   2> 1811588 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/clustering/carrot2/lingo-attributes.xml
   [junit4]   2> 1811591 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/clustering/carrot2/kmeans-attributes.xml
   [junit4]   2> 1811595 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/admin-extra.menu-bottom.html
   [junit4]   2> 1811598 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/currency.xml
   [junit4]   2> 1811603 T4544 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr
   [junit4]   2> 1811605 T4544 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/
   [junit4]   2> 1811606 T4544 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1811607 T4566 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1811607 T4566 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1811608 T4566 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1811609 T4567 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1811612 T4563 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "node_name":"188.138.97.18:8983_solr",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://188.138.97.18:8983/solr",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 1811612 T4563 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1811616 T4567 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1812609 T4566 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1812609 T4566 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1812610 T4566 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:collection1
   [junit4]   2> 1812610 T4566 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1812614 T4566 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1812615 T4566 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=collection1 specified config exists in ZooKeeper
   [junit4]   2> 1812615 T4566 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/'
   [junit4]   2> 1812651 T4566 oasc.SolrConfig.initLibs Adding specified lib dirs to ClassLoader
   [junit4]   2> 1812653 T4566 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/extraction/lib (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/extraction/lib).
   [junit4]   2> 1812654 T4566 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 1812654 T4566 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/clustering/lib/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/clustering/lib).
   [junit4]   2> 1812654 T4566 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 1812655 T4566 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/langid/lib/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/langid/lib).
   [junit4]   2> 1812655 T4566 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 1812656 T4566 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/velocity/lib (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/velocity/lib).
   [junit4]   2> 1812657 T4566 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 1812686 T4566 oasu.SolrIndexConfig.<init> IndexWriter infoStream solr logging is enabled
   [junit4]   2> 1812689 T4566 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1812753 T4566 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1812757 T4566 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1812765 T4566 oass.IndexSchema.readSchema [collection1] Schema name=example
   [junit4]   2> 1813125 T4566 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1813126 T4566 oasr.RestManager$Registry.registerManagedResource Registered ManagedResource impl org.apache.solr.rest.schema.analysis.ManagedWordSetResource for path /schema/analysis/stopwords/english
   [junit4]   2> 1813126 T4566 oasr.RestManager$Registry.registerManagedResource Registered ManagedResource impl org.apache.solr.rest.schema.analysis.ManagedSynonymFilterFactory$SynonymManager for path /schema/analysis/synonyms/english
   [junit4]   2> 1813146 T4566 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1813151 T4566 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1813288 T4566 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1813289 T4566 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1813289 T4566 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data/
   [junit4]   2> 1813289 T4566 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@42632047
   [junit4]   2> 1813290 T4566 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for newSearcher: org.apache.solr.core.QuerySenderListener{queries=[]}
   [junit4]   2> 1813290 T4566 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for firstSearcher: org.apache.solr.core.QuerySenderListener{queries=[{q=static firstSearcher warming in solrconfig.xml}]}
   [junit4]   2> 1813291 T4566 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@376f64c9
   [junit4]   2> 1813291 T4566 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 1813291 T4566 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1813291 T4566 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1813292 T4566 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 0 directories currently being tracked
   [junit4]   2> 1813292 T4566 oasc.CoreContainer.create ERROR Error creating core [collection1]: access denied ("java.io.FilePermission" "/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data" "write") org.apache.solr.common.SolrException: access denied ("java.io.FilePermission" "/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data" "write")
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:890)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data" "write")
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.NativeFSLockFactory.<init>(NativeFSLockFactory.java:93)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:502)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
   [junit4]   2> 	... 8 more
   [junit4]   2> 
   [junit4]   2> 1813294 T4544 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1813295 T4544 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1813296 T4570 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fb33d0d name:ZooKeeperConnection Watcher:127.0.0.1:42445 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1813296 T4544 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1813296 T4544 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1813298 T4544 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=907018964
   [junit4]   2> 1813298 T4544 oasc.Overseer.close Overseer (id=92449228915998721-188.138.97.18:8983_solr-n_0000000000) closing
   [junit4]   2> 1813298 T4563 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 188.138.97.18:8983_solr
   [junit4]   2> 1814799 T4571 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:347)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:344)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:344)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:346)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:296)
   [junit4]   2> 
   [junit4]   2> 1814799 T4567 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1814804 T4544 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=907018964
   [junit4]   2> 1814806 T4544 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:42445 42445
   [junit4]   2> 1815219 T4554 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:42445 42445
   [junit4]   2> 1815221 T4544 oas.SolrTestCaseJ4.tearDown ###Ending testChrootBootstrap
   [junit4]   2> 1815229 T4544 oas.SolrTestCaseJ4.setUp ###Starting testInitPathExists
   [junit4]   2> 1815230 T4544 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1815231 T4572 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1815331 T4544 oasc.ZkTestServer.run start zk server on port:48803
   [junit4]   2> 1815331 T4544 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1815332 T4544 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1815336 T4578 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5bd3cb1d name:ZooKeeperConnection Watcher:127.0.0.1:48803 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1815336 T4544 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1815337 T4544 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1815337 T4544 oascc.SolrZkClient.makePath makePath: /foo/bar4
   [junit4]   2> 1815342 T4544 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/'
   [junit4]   2> 1815360 T4544 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/solr.xml
   [junit4]   2> 1815374 T4544 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr
   [junit4]   2> 1815375 T4544 oasc.CoreContainer.<init> New CoreContainer 1111728605
   [junit4]   2> 1815375 T4544 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/]
   [junit4]   2> 1815375 T4544 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 1815376 T4544 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 1815376 T4544 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 1815376 T4544 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1815376 T4544 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1815376 T4544 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1815376 T4544 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1815377 T4544 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1815377 T4544 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1815378 T4544 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=0&connTimeout=0&retry=false
   [junit4]   2> 1815379 T4544 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1815379 T4544 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1815379 T4544 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 1815379 T4544 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:48803/foo/bar4
   [junit4]   2> 1815379 T4544 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1815379 T4544 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1815380 T4544 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1815381 T4580 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@51a7d81b name:ZooKeeperConnection Watcher:127.0.0.1:48803 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1815382 T4544 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1815382 T4544 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1815384 T4544 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1815385 T4582 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@797b18d3 name:ZooKeeperConnection Watcher:127.0.0.1:48803/foo/bar4 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1815385 T4544 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1815387 T4544 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1815389 T4544 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1815391 T4544 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1815394 T4544 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1815397 T4544 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1815400 T4544 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1815402 T4544 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/188.138.97.18:8983_solr
   [junit4]   2> 1815403 T4544 oascc.SolrZkClient.makePath makePath: /live_nodes/188.138.97.18:8983_solr
   [junit4]   2> 1815405 T4544 oascc.SolrZkClient.makePath makePath: /collections
   [junit4]   2> 1815407 T4544 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1815408 T4544 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1815409 T4544 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1815410 T4544 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 188.138.97.18:8983_solr
   [junit4]   2> 1815410 T4544 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1815411 T4544 oasc.Overseer.start Overseer (id=92449229189021698-188.138.97.18:8983_solr-n_0000000000) starting
   [junit4]   2> 1815412 T4544 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1815416 T4544 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1815416 T4584 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1815417 T4544 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1815418 T4544 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1815419 T4544 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1815420 T4544 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr
   [junit4]   2> 1815420 T4583 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1815422 T4544 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/
   [junit4]   2> 1815422 T4544 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1815423 T4544 oasc.ZkController.bootstrapConf bootstrapping config for 1 cores into ZooKeeper using solr.xml from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/
   [junit4]   2> 1815423 T4544 oasc.ZkController.bootstrapConf Uploading directory /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/conf with name collection1 for SolrCore collection1
   [junit4]   2> 1815423 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/spellings.txt
   [junit4]   2> 1815425 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/stopwords.txt
   [junit4]   2> 1815427 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/scripts.conf
   [junit4]   2> 1815429 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/admin-extra.html
   [junit4]   2> 1815430 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/schema.xml
   [junit4]   2> 1815432 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/solrconfig.xml
   [junit4]   2> 1815434 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/admin-extra.menu-top.html
   [junit4]   2> 1815435 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/elevate.xml
   [junit4]   2> 1815437 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/_schema_analysis_synonyms_english.json
   [junit4]   2> 1815438 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_nl.txt
   [junit4]   2> 1815441 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_eu.txt
   [junit4]   2> 1815443 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ga.txt
   [junit4]   2> 1815445 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_fr.txt
   [junit4]   2> 1815447 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_bg.txt
   [junit4]   2> 1815460 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stemdict_nl.txt
   [junit4]   2> 1815462 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/contractions_ca.txt
   [junit4]   2> 1815464 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ja.txt
   [junit4]   2> 1815465 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_id.txt
   [junit4]   2> 1815466 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_it.txt
   [junit4]   2> 1815467 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_fi.txt
   [junit4]   2> 1815469 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/contractions_fr.txt
   [junit4]   2> 1815470 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_el.txt
   [junit4]   2> 1815472 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_en.txt
   [junit4]   2> 1815473 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_da.txt
   [junit4]   2> 1815474 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stoptags_ja.txt
   [junit4]   2> 1815476 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_no.txt
   [junit4]   2> 1815477 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_pt.txt
   [junit4]   2> 1815478 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/hyphenations_ga.txt
   [junit4]   2> 1815479 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ru.txt
   [junit4]   2> 1815480 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ro.txt
   [junit4]   2> 1815482 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_cz.txt
   [junit4]   2> 1815483 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_hu.txt
   [junit4]   2> 1815484 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_th.txt
   [junit4]   2> 1815485 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ckb.txt
   [junit4]   2> 1815486 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_gl.txt
   [junit4]   2> 1815487 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ar.txt
   [junit4]   2> 1815488 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_tr.txt
   [junit4]   2> 1815490 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_es.txt
   [junit4]   2> 1815491 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/userdict_ja.txt
   [junit4]   2> 1815492 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/contractions_it.txt
   [junit4]   2> 1815493 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ca.txt
   [junit4]   2> 1815495 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_lv.txt
   [junit4]   2> 1815496 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_de.txt
   [junit4]   2> 1815497 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/contractions_ga.txt
   [junit4]   2> 1815499 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_hi.txt
   [junit4]   2> 1815500 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_hy.txt
   [junit4]   2> 1815501 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_sv.txt
   [junit4]   2> 1815502 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_fa.txt
   [junit4]   2> 1815504 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/synonyms.txt
   [junit4]   2> 1815506 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/update-script.js
   [junit4]   2> 1815508 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facet_pivot.vm
   [junit4]   2> 1815509 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/jquery.autocomplete.css
   [junit4]   2> 1815510 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/debug.vm
   [junit4]   2> 1815511 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/query_form.vm
   [junit4]   2> 1815513 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/pagination_top.vm
   [junit4]   2> 1815514 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/did_you_mean.vm
   [junit4]   2> 1815515 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/query_spatial.vm
   [junit4]   2> 1815516 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/hit_plain.vm
   [junit4]   2> 1815517 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/pagination_bottom.vm
   [junit4]   2> 1815519 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/header.vm
   [junit4]   2> 1815520 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/layout.vm
   [junit4]   2> 1815521 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/suggest.vm
   [junit4]   2> 1815522 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/README.txt
   [junit4]   2> 1815523 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/head.vm
   [junit4]   2> 1815524 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/footer.vm
   [junit4]   2> 1815526 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/results_list.vm
   [junit4]   2> 1815527 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/jquery.autocomplete.js
   [junit4]   2> 1815528 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/hit_grouped.vm
   [junit4]   2> 1815529 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facet_fields.vm
   [junit4]   2> 1815531 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/query_group.vm
   [junit4]   2> 1815532 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/query.vm
   [junit4]   2> 1815533 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/cluster_results.vm
   [junit4]   2> 1815534 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/VM_global_library.vm
   [junit4]   2> 1815535 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facet_queries.vm
   [junit4]   2> 1815537 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/mime_type_lists.vm
   [junit4]   2> 1815538 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/error.vm
   [junit4]   2> 1815539 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/product_doc.vm
   [junit4]   2> 1815540 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/main.css
   [junit4]   2> 1815541 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/browse.vm
   [junit4]   2> 1815543 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/richtext_doc.vm
   [junit4]   2> 1815544 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/cluster.vm
   [junit4]   2> 1815545 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facets.vm
   [junit4]   2> 1815546 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/tabs.vm
   [junit4]   2> 1815547 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/join_doc.vm
   [junit4]   2> 1815549 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/hit.vm
   [junit4]   2> 1815550 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facet_ranges.vm
   [junit4]   2> 1815551 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1815552 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/_schema_analysis_stopwords_english.json
   [junit4]   2> 1815553 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/protwords.txt
   [junit4]   2> 1815554 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/_rest_managed.json
   [junit4]   2> 1815555 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/example.xsl
   [junit4]   2> 1815557 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/example_rss.xsl
   [junit4]   2> 1815558 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/updateXml.xsl
   [junit4]   2> 1815559 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/luke.xsl
   [junit4]   2> 1815560 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/example_atom.xsl
   [junit4]   2> 1815561 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/mapping-FoldToASCII.txt
   [junit4]   2> 1815563 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/clustering/carrot2/stc-attributes.xml
   [junit4]   2> 1815565 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/clustering/carrot2/lingo-attributes.xml
   [junit4]   2> 1815566 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/clustering/carrot2/kmeans-attributes.xml
   [junit4]   2> 1815567 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/admin-extra.menu-bottom.html
   [junit4]   2> 1815568 T4544 oascc.SolrZkClient.makePath makePath: /configs/collection1/currency.xml
   [junit4]   2> 1815571 T4544 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr
   [junit4]   2> 1815572 T4544 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/
   [junit4]   2> 1815572 T4544 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1815573 T4586 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1815574 T4586 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1815575 T4586 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1815575 T4587 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1815577 T4583 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "node_name":"188.138.97.18:8983_solr",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://188.138.97.18:8983/solr",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 1815578 T4583 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1815580 T4587 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1816575 T4586 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1816575 T4586 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1816576 T4586 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:collection1
   [junit4]   2> 1816577 T4586 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1816580 T4586 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1816581 T4586 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=collection1 specified config exists in ZooKeeper
   [junit4]   2> 1816581 T4586 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/'
   [junit4]   2> 1816612 T4586 oasc.SolrConfig.initLibs Adding specified lib dirs to ClassLoader
   [junit4]   2> 1816612 T4586 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/extraction/lib (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/extraction/lib).
   [junit4]   2> 1816612 T4586 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 1816613 T4586 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/clustering/lib/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/clustering/lib).
   [junit4]   2> 1816613 T4586 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 1816613 T4586 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/langid/lib/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/langid/lib).
   [junit4]   2> 1816613 T4586 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 1816614 T4586 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/velocity/lib (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/velocity/lib).
   [junit4]   2> 1816614 T4586 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 1816637 T4586 oasu.SolrIndexConfig.<init> IndexWriter infoStream solr logging is enabled
   [junit4]   2> 1816639 T4586 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1816692 T4586 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1816694 T4586 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1816700 T4586 oass.IndexSchema.readSchema [collection1] Schema name=example
   [junit4]   2> 1817043 T4586 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1817045 T4586 oasr.RestManager$Registry.registerManagedResource Registered ManagedResource impl org.apache.solr.rest.schema.analysis.ManagedWordSetResource for path /schema/analysis/stopwords/english
   [junit4]   2> 1817045 T4586 oasr.RestManager$Registry.registerManagedResource Registered ManagedResource impl org.apache.solr.rest.schema.analysis.ManagedSynonymFilterFactory$SynonymManager for path /schema/analysis/synonyms/english
   [junit4]   2> 1817060 T4586 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1817062 T4586 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1817121 T4586 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1817121 T4586 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1817121 T4586 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data/
   [junit4]   2> 1817122 T4586 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@42632047
   [junit4]   2> 1817122 T4586 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for newSearcher: org.apache.solr.core.QuerySenderListener{queries=[]}
   [junit4]   2> 1817122 T4586 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for firstSearcher: org.apache.solr.core.QuerySenderListener{queries=[{q=static firstSearcher warming in solrconfig.xml}]}
   [junit4]   2> 1817123 T4586 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@399c6914
   [junit4]   2> 1817123 T4586 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 1817123 T4586 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1817123 T4586 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1817123 T4586 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 0 directories currently being tracked
   [junit4]   2> 1817123 T4586 oasc.CoreContainer.create ERROR Error creating core [collection1]: access denied ("java.io.FilePermission" "/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data" "write") org.apache.solr.common.SolrException: access denied ("java.io.FilePermission" "/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data" "write")
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:890)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/data" "write")
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.NativeFSLockFactory.<init>(NativeFSLockFactory.java:93)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:502)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
   [junit4]   2> 	... 8 more
   [junit4]   2> 
   [junit4]   2> 1817127 T4544 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1111728605
   [junit4]   2> 1817127 T4544 oasc.Overseer.close Overseer (id=92449229189021698-188.138.97.18:8983_solr-n_0000000000) closing
   [junit4]   2> 1817127 T4583 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 188.138.97.18:8983_solr
   [junit4]   2> 1818628 T4589 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:347)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:344)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:344)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:346)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:296)
   [junit4]   2> 
   [junit4]   2> 1818633 T4587 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1818635 T4544 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1111728605
   [junit4]   2> 1818637 T4544 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:48803 48803
   [junit4]   2> 1819315 T4572 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:48803 48803
   [junit4]   2> 1819318 T4544 oas.SolrTestCaseJ4.tearDown ###Ending testInitPathExists
   [junit4]   2> 1819326 T4544 oas.SolrTestCaseJ4.setUp ###Starting testWithUploadDir
   [junit4]   2> 1819328 T4544 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1819329 T4590 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1819428 T4544 oasc.ZkTestServer.run start zk server on port:43092
   [junit4]   2> 1819429 T4544 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1819430 T4544 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1819433 T4596 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2303555d name:ZooKeeperConnection Watcher:127.0.0.1:43092 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1819434 T4544 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2>

[...truncated too long message...]

 [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #10: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=7043, name=Thread-2909, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([D93C69EAE6E4DD07]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #11: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=7042, name=Thread-2908, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([D93C69EAE6E4DD07]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 2682183 T7032 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{}, sim=DefaultSimilarity, locale=es_PY, timezone=Mideast/Riyadh88
   [junit4]   2> NOTE: Linux 3.13.0-35-generic amd64/Oracle Corporation 1.7.0_67 (64-bit)/cpus=8,threads=1,free=248207472,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [QueryElevationComponentTest, TestUtils, TestCloudManagedSchemaConcurrent, DeleteReplicaTest, TestShardHandlerFactory, TestImplicitCoreProperties, CollectionsAPIDistributedZkTest, TestLazyCores, ConvertedLegacyTest, SolrRequestParserTest, TestRandomMergePolicy, DistributedExpandComponentTest, AliasIntegrationTest, TestCollationField, QueryParsingTest, RequestHandlersTest, CircularListTest, TestQueryUtils, TestFieldTypeCollectionResource, TestChildDocTransformer, ShardRoutingCustomTest, PrimUtilsTest, VMParamsZkACLAndCredentialsProvidersTest, DistributedQueryComponentOptimizationTest, DistributedQueryComponentCustomSortTest, TestOmitPositions, CSVRequestHandlerTest, HdfsDirectoryTest, TestNRTOpen, DateMathParserTest, TestQuerySenderNoQuery, TermVectorComponentDistributedTest, TestBlendedInfixSuggestions, MultiThreadedOCPTest, TestFunctionQuery, HdfsUnloadDistributedZkTest, TestSolr4Spatial2, TestComplexPhraseQParserPlugin, CurrencyFieldXmlFileTest, RollingRestartTest, TestTolerantSearch, TestPivotHelperCode, IndexSchemaRuntimeFieldTest, TestStressLucene, TestSolrQueryParser, DocValuesTest, SolrIndexConfigTest, TestRestManager, TestCoreContainer, HardAutoCommitTest, FullSolrCloudDistribCmdsTest, UniqFieldsUpdateProcessorFactoryTest, DirectUpdateHandlerOptimizeTest, BJQParserTest, BlockDirectoryTest, ShardSplitTest, LegacyHTMLStripCharFilterTest, TestLFUCache, RangeFacetTest, TestManagedSchemaFieldResource, CoreContainerCoreInitFailuresTest, AnalyticsQueryTest, TestPHPSerializedResponseWriter, OpenCloseCoreStressTest, SearchHandlerTest, TriLevelCompositeIdRoutingTest, TestShortCircuitedRequests, PreAnalyzedFieldTest, OverriddenZkACLAndCredentialsProvidersTest, RAMDirectoryFactoryTest, URLClassifyProcessorTest, TestXIncludeConfig, DocExpirationUpdateProcessorFactoryTest, CopyFieldTest, TestDistributedGrouping, TestFieldSortValues, TestReplicationHandlerBackup, AnalyticsMergeStrategyTest, DateFieldTest, UpdateRequestProcessorFactoryTest, DebugComponentTest, PolyFieldTest, TestCSVResponseWriter, BasicDistributedZk2Test, TestQueryTypes, TestInfoStreamLogging, CoreMergeIndexesAdminHandlerTest, TestCloudPivotFacet, NumericFieldsTest, TestBinaryField, TestAtomicUpdateErrorCases, TestConfigSets, TestSolrXmlPersistor, TestManagedSynonymFilterFactory, TestZkChroot, SliceStateTest, TestIndexingPerformance, SoftAutoCommitTest, RankQueryTest, TestBadConfig, DirectSolrConnectionTest, TestRecovery, TestUpdate, TestNonNRTOpen, TestReplicationHandler, TestMultiCoreConfBootstrap, TestReload, TestRangeQuery, SolrCoreTest, StatsComponentTest, TestGroupingSearch, SolrCmdDistributorTest, DirectUpdateHandlerTest, DistributedQueryElevationComponentTest, SchemaVersionSpecificBehaviorTest, TestPseudoReturnFields, TestWordDelimiterFilterFactory, StatelessScriptUpdateProcessorFactoryTest, TestSolrDeletionPolicy1, CacheHeaderTest, DocumentAnalysisRequestHandlerTest, FileBasedSpellCheckerTest, TestValueSourceCache, TermVectorComponentTest, LoggingHandlerTest, SolrPluginUtilsTest, JSONWriterTest, BinaryUpdateRequestHandlerTest, HighlighterConfigTest, TestQuerySenderListener, TestStressRecovery, ExpressionTest, FieldFacetExtrasTest, FieldFacetTest, QueryFacetTest, FunctionTest, CustomCollectionTest, DeleteInactiveReplicaTest, DistribDocExpirationUpdateProcessorTest, OutOfBoxZkACLAndCredentialsProvidersTest, OverseerRolesTest, ReplicationFactorTest, SSLMigrationTest, SharedFSAutoReplicaFailoverTest, SharedFSAutoReplicaFailoverUtilsTest, TestDistribDocBasedVersion, TestModifyConfFiles, HdfsBasicDistributedZk2Test, HdfsBasicDistributedZkTest, StressHdfsTest, CachingDirectoryFactoryTest]
   [junit4] Completed on J1 in 8.10s, 1 test, 1 error <<< FAILURES!

[...truncated 193 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:471: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:60: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1358: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:961: There were test failures: 438 suites, 1830 tests, 3 suite-level errors, 2 errors, 45 ignored (23 assumptions)

Total time: 89 minutes 29 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.7.0_67 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_40-ea-b04) - Build # 11240 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11240/
Java: 64bit/jdk1.8.0_40-ea-b04 -XX:+UseCompressedOops -XX:+UseG1GC

7 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestZkChroot

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=5251, name=coreLoadExecutor-2568-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=5251, name=coreLoadExecutor-2568-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
	at java.lang.Thread.getStackTrace(Thread.java:1552)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:729)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1272)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1163)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1155)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@3373529f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@48072962)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:699)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:696)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:712)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	... 1 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestZkChroot

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=5276, name=coreLoadExecutor-2578-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=5276, name=coreLoadExecutor-2578-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
	at java.lang.Thread.getStackTrace(Thread.java:1552)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:729)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1272)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1163)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1155)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@236306cd lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@260b93d5)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:699)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:696)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:712)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	... 1 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestZkChroot

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=5296, name=coreLoadExecutor-2585-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=5296, name=coreLoadExecutor-2585-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
	at java.lang.Thread.getStackTrace(Thread.java:1552)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:729)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1272)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1163)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1155)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@64998627 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@61e1f9d7)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:699)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:696)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:712)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	... 1 more


REGRESSION:  org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.testDistribSearch

Error Message:
No live SolrServers available to handle this request:[https://127.0.0.1:39925/kwy, https://127.0.0.1:52356/kwy, https://127.0.0.1:55432/kwy]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:39925/kwy, https://127.0.0.1:52356/kwy, https://127.0.0.1:55432/kwy]
	at __randomizedtesting.SeedInfo.seed([DF1F884DADB46034:5EF90655DAEB0008]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:322)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:874)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:658)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601)
	at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.removeAndWaitForLastReplicaGone(DeleteLastCustomShardedReplicaTest.java:116)
	at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.doTest(DeleteLastCustomShardedReplicaTest.java:106)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not  remove replica : customcollreplicadeletion/a/core_node1
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:550)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:343)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:304)
	... 46 more


REGRESSION:  org.apache.solr.cloud.DeleteReplicaTest.testDistribSearch

Error Message:
No live SolrServers available to handle this request:[https://127.0.0.1:41348, https://127.0.0.1:45666, https://127.0.0.1:50256, https://127.0.0.1:41428, https://127.0.0.1:37880]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:41348, https://127.0.0.1:45666, https://127.0.0.1:50256, https://127.0.0.1:41428, https://127.0.0.1:37880]
	at __randomizedtesting.SeedInfo.seed([DF1F884DADB46034:5EF90655DAEB0008]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:322)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:874)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:658)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601)
	at org.apache.solr.cloud.DeleteReplicaTest.removeAndWaitForReplicaGone(DeleteReplicaTest.java:139)
	at org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:123)
	at org.apache.solr.cloud.DeleteReplicaTest.doTest(DeleteReplicaTest.java:87)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not  remove replica : delLiveColl/shard2/core_node3
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:550)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:343)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:304)
	... 47 more


REGRESSION:  org.apache.solr.core.CachingDirectoryFactoryTest.stressTest

Error Message:
Captured an uncaught exception in thread: Thread[id=6935, name=Thread-2497, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=6935, name=Thread-2497, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path11/7/17" "write")
	at __randomizedtesting.SeedInfo.seed([DF1F884DADB46034]:0)
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
	at java.security.AccessController.checkPermission(AccessController.java:884)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
	at java.nio.file.Files.createDirectory(Files.java:674)
	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
	at java.nio.file.Files.createDirectories(Files.java:727)
	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)


REGRESSION:  org.apache.solr.core.RAMDirectoryFactoryTest.test

Error Message:
access denied ("java.io.FilePermission" "/fake/path" "write")

Stack Trace:
java.security.AccessControlException: access denied ("java.io.FilePermission" "/fake/path" "write")
	at __randomizedtesting.SeedInfo.seed([DF1F884DADB46034:574BB79703480DCC]:0)
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
	at java.security.AccessController.checkPermission(AccessController.java:884)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
	at java.nio.file.Files.createDirectory(Files.java:674)
	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
	at java.nio.file.Files.createDirectories(Files.java:727)
	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
	at org.apache.solr.core.RAMDirectoryFactoryTest.dotestOpenReturnsTheSameForSamePath(RAMDirectoryFactoryTest.java:46)
	at org.apache.solr.core.RAMDirectoryFactoryTest.test(RAMDirectoryFactoryTest.java:33)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11237 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-DF1F884DADB46034-001/init-core-data-001
   [junit4]   2> 70593 T223 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 70593 T223 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /kwy/
   [junit4]   2> 70599 T223 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 70599 T223 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 70600 T224 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 70700 T223 oasc.ZkTestServer.run start zk server on port:57353
   [junit4]   2> 70700 T223 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 70701 T223 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 70704 T230 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@35ce8614 name:ZooKeeperConnection Watcher:127.0.0.1:57353 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 70705 T223 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 70705 T223 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 70705 T223 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 70712 T223 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 70713 T223 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 70716 T232 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5e9c7ddb name:ZooKeeperConnection Watcher:127.0.0.1:57353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 70716 T223 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 70716 T223 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 70716 T223 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 70719 T223 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 70720 T223 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 70721 T223 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 70723 T223 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 70723 T223 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 70726 T223 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 70726 T223 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 70728 T223 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 70729 T223 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 70730 T223 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 70730 T223 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 70732 T223 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 70733 T223 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 70734 T223 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 70734 T223 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 70736 T223 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 70736 T223 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 70737 T223 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 70738 T223 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 70739 T223 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 70739 T223 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 70741 T223 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 70741 T223 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 70742 T223 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 70743 T223 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 70747 T223 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 70747 T223 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 70749 T234 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@680fdb51 name:ZooKeeperConnection Watcher:127.0.0.1:57353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 70749 T223 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 70750 T223 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 71144 T223 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 71148 T223 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 71166 T223 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:?????
   [junit4]   2> 71168 T223 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 71168 T223 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 71169 T223 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-DF1F884DADB46034-001/tempDir-002
   [junit4]   2> 71169 T223 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-DF1F884DADB46034-001/tempDir-002/'
   [junit4]   2> 71185 T223 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-DF1F884DADB46034-001/tempDir-002/solr.xml
   [junit4]   2> 71211 T223 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-DF1F884DADB46034-001/tempDir-002
   [junit4]   2> 71211 T223 oasc.CoreContainer.<init> New CoreContainer 527058626
   [junit4]   2> 71212 T223 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-DF1F884DADB46034-001/tempDir-002/]
   [junit4]   2> 71212 T223 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 71213 T223 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 71213 T223 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 71213 T223 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 71213 T223 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 71214 T223 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 71214 T223 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 71214 T223 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 71214 T223 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 71216 T223 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 71217 T223 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 71217 T223 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 71217 T223 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 71217 T223 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57353/solr
   [junit4]   2> 71218 T223 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 71218 T223 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 71218 T223 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 71220 T245 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@20c5b241 name:ZooKeeperConnection Watcher:127.0.0.1:57353 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 71220 T223 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 71221 T223 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 71222 T223 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 71224 T247 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e92d86a name:ZooKeeperConnection Watcher:127.0.0.1:57353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 71224 T223 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 71225 T223 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 71227 T223 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 71228 T223 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 71230 T223 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 71232 T223 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 71234 T223 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 71235 T223 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55432_kwy
   [junit4]   2> 71236 T223 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55432_kwy
   [junit4]   2> 71237 T223 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 71238 T223 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 71239 T223 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 71241 T223 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:55432_kwy
   [junit4]   2> 71241 T223 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 71242 T223 oasc.Overseer.start Overseer (id=92448366919024644-127.0.0.1:55432_kwy-n_0000000000) starting
   [junit4]   2> 71243 T223 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 71248 T223 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 71249 T249 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 71250 T223 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 71251 T223 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 71253 T223 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 71255 T248 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 71257 T223 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-DF1F884DADB46034-001/tempDir-002
   [junit4]   2> 71268 T223 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-DF1F884DADB46034-001/tempDir-002/conf/
   [junit4]   2> 71414 T223 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 71415 T223 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 71415 T223 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 71418 T223 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 71419 T223 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 71420 T253 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4c1b2886 name:ZooKeeperConnection Watcher:127.0.0.1:57353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 71420 T223 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 71421 T223 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 71422 T223 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 71424 T223 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 71424 T223 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 71424 T223 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 71425 T223 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 71426 T255 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@38b9ac4c name:ZooKeeperConnection Watcher:127.0.0.1:57353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 71426 T223 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 71427 T223 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 71429 T256 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 71432 T248 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: collection1
   [junit4]   2> 71432 T248 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 71432 T248 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 2
   [junit4]   2> 71434 T248 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/collection1/state.json
   [junit4]   2> 71435 T256 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 71436 T257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 71776 T223 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 71777 T223 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 71781 T223 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 71783 T223 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:?????
   [junit4]   2> 71785 T223 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 71785 T223 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 71786 T223 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-DF1F884DADB46034-001/tempDir-003
   [junit4]   2> 71786 T223 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-DF1F884DADB46034-001/tempDir-003/'
   [junit4]   2> 71802 T223 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-DF1F884DADB46034-001/tempDir-003/solr.xml
   [junit4]   2> 71808 T223 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-DF1F884DADB46034-001/tempDir-003
   [junit4]   2> 71809 T223 oasc.CoreContainer.<init> New CoreContainer 294460718
   [junit4]   2> 71809 T223 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-DF1F884DADB46034-001/tempDir-003/]
   [junit4]   2> 71809 T223 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 71810 T223 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 71810 T223 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 71810 T223 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 71810 T223 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 71811 T223 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 71811 T223 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 71811 T223 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 71811 T223 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 71813 T223 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 71814 T223 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 71814 T223 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 71814 T223 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 71815 T223 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57353/solr
   [junit4]   2> 71815 T223 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 71815 T223 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 71816 T223 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 71817 T268 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c8d026a name:ZooKeeperConnection Watcher:127.0.0.1:57353 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 71818 T223 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 71818 T223 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 71823 T223 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 71825 T270 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9e81aaa name:ZooKeeperConnection Watcher:127.0.0.1:57353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 71825 T223 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 71829 T223 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 72832 T223 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52356_kwy
   [junit4]   2> 72833 T223 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52356_kwy
   [junit4]   2> 72837 T223 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 72839 T223 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-DF1F884DADB46034-001/tempDir-003
   [junit4]   2> 72846 T223 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-DF1F884DADB46034-001/tempDir-003/conf/
   [junit4]   2> 72962 T223 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 72963 T223 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 72964 T223 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 73209 T223 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 73210 T223 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 73213 T223 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 73214 T223 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:?????
   [junit4]   2> 73216 T223 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 73216 T223 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 73216 T223 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-DF1F884DADB46034-001/tempDir-004
   [junit4]   2> 73216 T223 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-DF1F884DADB46034-001/tempDir-004/'
   [junit4]   2> 73227 T223 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-DF1F884DADB46034-001/tempDir-004/solr.xml
   [junit4]   2> 73233 T223 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-DF1F884DADB46034-001/tempDir-004
   [junit4]   2> 73234 T223 oasc.CoreContainer.<init> New CoreContainer 1771769262
   [junit4]   2> 73234 T223 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-DF1F884DADB46034-001/tempDir-004/]
   [junit4]   2> 73235 T223 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 73235 T223 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 73235 T223 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 73235 T223 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 73235 T223 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 73236 T223 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 73236 T223 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 73236 T223 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 73236 T223 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 73238 T223 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 73238 T223 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 73239 T223 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 73239 T223 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 73239 T223 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57353/solr
   [junit4]   2> 73239 T223 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 73240 T223 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 73240 T223 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 73242 T283 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7fef3ca7 name:ZooKeeperConnection Watcher:127.0.0.1:57353 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 73242 T223 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 73243 T223 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 73245 T223 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 73246 T285 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1aa4cee9 name:ZooKeeperConnection Watcher:127.0.0.1:57353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 73246 T223 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 73250 T223 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 74254 T223 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39925_kwy
   [junit4]   2> 74255 T223 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39925_kwy
   [junit4]   2> 74259 T223 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 74261 T223 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-DF1F884DADB46034-001/tempDir-004
   [junit4]   2> 74266 T223 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-DF1F884DADB46034-001/tempDir-004/conf/
   [junit4]   2> 74374 T223 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 74375 T223 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 74375 T223 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 74378 T223 oasc.AbstractFullDistribZkTestBase.createCollection Creating collection with stateFormat=2: customcollreplicadeletion
   [junit4]   2> 74378 T223 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 74379 T223 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 74380 T289 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11e110c0 name:ZooKeeperConnection Watcher:127.0.0.1:57353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 74380 T223 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 74381 T223 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 74382 T223 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 74451 T281 oasha.CollectionsHandler.handleCreateAction Creating Collection : replicationFactor=1&shards=a,b&maxShardsPerNode=5&router.name=implicit&name=customcollreplicadeletion&action=CREATE&numShards=1&stateFormat=2&wt=javabin&version=2
   [junit4]   2> 74452 T256 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 74453 T249 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"customcollreplicadeletion",
   [junit4]   2> 	  "replicationFactor":"1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "maxShardsPerNode":"5",
   [junit4]   2> 	  "shards":"a,b",
   [junit4]   2> 	  "stateFormat":"2",
   [junit4]   2> 	  "router.name":"implicit"}
   [junit4]   2> 74454 T290 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"customcollreplicadeletion",
   [junit4]   2> 	  "replicationFactor":"1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "maxShardsPerNode":"5",
   [junit4]   2> 	  "shards":"a,b",
   [junit4]   2> 	  "stateFormat":"2",
   [junit4]   2> 	  "router.name":"implicit"}
   [junit4]   2> 74455 T290 oasc.OverseerCollectionProcessor.createConfNode Only one config set found in zk - using it:conf1
   [junit4]   2> 74455 T290 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/customcollreplicadeletion 
   [junit4]   2> 74456 T290 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion
   [junit4]   2> 74459 T256 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 74461 T248 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: customcollreplicadeletion
   [junit4]   2> 74461 T248 oasc.Overseer$ClusterStateUpdater.createCollection Create collection customcollreplicadeletion with shards [a, b]
   [junit4]   2> 74461 T248 oasc.Overseer$ClusterStateUpdater.createCollection state version customcollreplicadeletion 2
   [junit4]   2> 74463 T248 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 74464 T257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 74464 T271 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 74464 T291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 74464 T256 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 74464 T286 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 74559 T290 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection customcollreplicadeletion, shardNames [a, b] , replicationFactor : 1
   [junit4]   2> 74559 T290 oasc.OverseerCollectionProcessor.createCollection Creating shard customcollreplicadeletion_a_replica1 as part of slice a of collection customcollreplicadeletion on 127.0.0.1:55432_kwy
   [junit4]   2> 74560 T290 oasc.OverseerCollectionProcessor.createCollection Creating shard customcollreplicadeletion_b_replica1 as part of slice b of collection customcollreplicadeletion on 127.0.0.1:39925_kwy
   [junit4]   2> 74636 T278 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_b_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=b&wt=javabin&version=2
   [junit4]   2> 74639 T240 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_a_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=a&wt=javabin&version=2
   [junit4]   2> 74641 T240 oasc.ZkController.publish publishing core=customcollreplicadeletion_a_replica1 state=down collection=customcollreplicadeletion
   [junit4]   2> 74642 T256 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 74643 T240 oasc.ZkController.preRegister Registering watch for external collection customcollreplicadeletion
   [junit4]   2> 74644 T278 oasc.ZkController.publish publishing core=customcollreplicadeletion_b_replica1 state=down collection=customcollreplicadeletion
   [junit4]   2> 74644 T248 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"customcollreplicadeletion_a_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:55432/kwy",
   [junit4]   2> 	  "node_name":"127.0.0.1:55432_kwy",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"a",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 74645 T278 oasc.ZkController.preRegister Registering watch for external collection customcollreplicadeletion
   [junit4]   2> 74646 T278 oascc.ZkStateReader.addZkWatch addZkWatch customcollreplicadeletion
   [junit4]   2> 74646 T278 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/customcollreplicadeletion/state.json from ZooKeeper... 
   [junit4]   2> 74647 T278 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 0 
   [junit4]   2> 74647 T278 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 74649 T248 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 74649 T286 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 74650 T286 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 1 
   [junit4]   2> 74650 T240 oascc.ZkStateReader.addZkWatch addZkWatch customcollreplicadeletion
   [junit4]   2> 74651 T240 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/customcollreplicadeletion/state.json from ZooKeeper... 
   [junit4]   2> 74652 T240 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 1 
   [junit4]   2> 74652 T240 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 74652 T240 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for customcollreplicadeletion_a_replica1
   [junit4]   2> 74652 T240 oasc.ZkController.createCollectionZkNode Check for collection zkNode:customcollreplicadeletion
   [junit4]   2> 74653 T240 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 74653 T240 oascc.ZkStateReader.readConfigName Load collection config from:/collections/customcollreplicadeletion
   [junit4]   2> 74653 T248 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 1 
   [junit4]   2> 74654 T240 oascc.ZkStateReader.readConfigName path=/collections/customcollreplicadeletion configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 74654 T248 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"customcollreplicadeletion_b_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:39925/kwy",
   [junit4]   2> 	  "node_name":"127.0.0.1:39925_kwy",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"b",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 74654 T240 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-DF1F884DADB46034-001/tempDir-002/customcollreplicadeletion_a_replica1/'
   [junit4]   2> 74655 T256 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 74655 T248 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 74656 T286 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 74656 T286 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 2 
   [junit4]   2> 74656 T256 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 74658 T256 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 2 
   [junit4]   2> 74678 T240 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 74691 T240 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 74692 T240 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 74701 T240 oass.IndexSchema.readSchema [customcollreplicadeletion_a_replica1] Schema name=test
   [junit4]   2> 74820 T240 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 74826 T240 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 74828 T240 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 74844 T240 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 74848 T240 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 74851 T240 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 74852 T240 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 74853 T240 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 74854 T240 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 74854 T240 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 74855 T240 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 74855 T240 oasc.CoreContainer.create Creating SolrCore 'customcollreplicadeletion_a_replica1' using configuration from collection customcollreplicadeletion
   [junit4]   2> 74855 T240 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 74856 T240 oasc.SolrCore.<init> [customcollreplicadeletion_a_replica1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-DF1F884DADB46034-001/tempDir-002/customcollreplicadeletion_a_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-DF1F884DADB46034-001/tempDir-002/customcollreplicadeletion_a_replica1/data/
   [junit4]   2> 74856 T240 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1049db4e
   [junit4]   2> 74857 T240 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-DF1F884DADB46034-001/tempDir-002/customcollreplicadeletion_a_replica1/data
   [junit4]   2> 74860 T240 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-DF1F884DADB46034-001/tempDir-002/customcollreplicadeletion_a_replica1/data/index/
   [junit4]   2> 74860 T240 oasc.SolrCore.initIndex WARN [customcollreplicadeletion_a_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-DF1F884DADB46034-001/tempDir-002/customcollreplicadeletion_a_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 74861 T240 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-DF1F884DADB46034-001/tempDir-002/customcollreplicadeletion_a_replica1/data/index
   [junit4]   2> 74861 T240 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=48.4775390625, floorSegmentMB=0.4384765625, forceMergeDeletesPctAllowed=0.12369125592327168, segmentsPerTier=23.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2772990419729281
   [junit4]   2> 74862 T240 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@47b1adf lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2bf2ac0b),segFN=segments_1,generation=1}
   [junit4]   2> 74863 T240 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 74870 T240 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 74870 T240 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 74871 T240 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 74871 T240 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 74871 T240 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 74872 T240 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 74872 T240 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 74873 T240 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 74873 T240 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 74874 T240 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 74875 T240 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 74875 T240 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 74876 T240 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 74877 T240 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 74877 T240 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 74877 T240 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 74878 T240 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 74878 T240 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 74879 T240 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 74880 T240 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 74882 T240 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 74883 T240 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 74884 T240 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 74885 T240 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 74887 T240 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 74887 T240 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 74888 T240 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=17, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 74888 T240 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@47b1adf lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2bf2ac0b),segFN=segments_1,generation=1}
   [junit4]   2> 74889 T240 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 74889 T240 oass.SolrIndexSearcher.<init> Opening Searcher@4caabe19[customcollreplicadeletion_a_replica1] main
   [junit4]   2> 74889 T240 oascc.ZkStateReader.readConfigName Load collection config from:/collections/customcollreplicadeletion
   [junit4]   2> 74894 T240 oascc.ZkStateReader.readConfigName path=/collections/customcollreplicadeletion configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 74895 T240 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 74895 T240 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 74896 T240 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 74896 T240 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 74897 T240 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 74897 T240 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 74899 T240 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 74902 T240 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 74902 T240 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 74903 T240 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 74908 T294 oasc.SolrCore.registerSearcher [customcollreplicadeletion_a_replica1] Registered new searcher Searcher@4caabe19[customcollreplicadeletion_a_replica1] main{UninvertingDirectoryReader()}
   [junit4]   2> 74916 T240 oasc.CoreContainer.registerCore registering core: customcollreplicadeletion_a_replica1
   [junit4]   2> 74917 T240 oasc.ZkController.register Register replica - core:customcollreplicadeletion_a_replica1 address:https://127.0.0.1:55432/kwy collection:customcollreplicadeletion shard:a
   [junit4]   2> 74918 T240 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion/leader_elect/a/election
   [junit4]   2> 74923 T240 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard a
   [junit4]   2> 74924 T256 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 74925 T240 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 74925 T240 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C18 name=customcollreplicadeletion_a_replica1 org.apache.solr.core.SolrCore@52fb267f url=https://127.0.0.1:55432/kwy/customcollreplicadeletion_a_replica1 node=127.0.0.1:55432_kwy C18_STATE=coll:customcollreplicadeletion core:customcollreplicadeletion_a_replica1 props:{core=customcollreplicadeletion_a_replica1, base_url=https://127.0.0.1:55432/kwy, node_name=127.0.0.1:55432_kwy, state=down}
   [junit4]   2> 74925 T240 C18 P55432 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:55432/kwy/customcollreplicadeletion_a_replica1/
   [junit4]   2> 74926 T240 C18 P55432 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 74926 T240 C18 P55432 oasc.SyncStrategy.syncToMe https://127.0.0.1:55432/kwy/customcollreplicadeletion_a_replica1/ has no replicas
   [junit4]   2> 74927 T240 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:55432/kwy/customcollreplicadeletion_a_replica1/ a
   [junit4]   2> 74927 T240 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion/leaders/a
   [junit4]   2> 74931 T248 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 2 
   [junit4]   2> 74934 T248 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 74935 T256 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 74935 T286 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 74936 T286 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 3 
   [junit4]   2> 74936 T256 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 3 
   [junit4]   2> 74938 T248 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 3 
   [junit4]   2> 74940 T256 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 74940 T248 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 74941 T286 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 74941 T256 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 74942 T286 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 4 
   [junit4]   2> 74943 T256 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 4 
   [junit4]   2> 74985 T240 oasc.ZkController.register We are https://127.0.0.1:55432/kwy/customcollreplicadeletion_a_replica1/ and leader is https://127.0.0.1:55432/kwy/customcollreplicadeletion_a_replica1/
   [junit4]   2> 74986 T240 oasc.ZkController.register No LogReplay needed for core=customcollreplicadeletion_a_replica1 baseURL=https://127.0.0.1:55432/kwy
   [junit4]   2> 74986 T240 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 74986 T240 oasc.ZkController.publish publishing core=customcollreplicadeletion_a_replica1 state=active collection=customcollreplicadeletion
   [junit4]   2> 74988 T256 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 74992 T240 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 4 
   [junit4]   2> 74993 T240 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_a_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=a&wt=javabin&version=2} status=0 QTime=354 
   [junit4]   2> 74995 T248 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 4 
   [junit4]   2> 74996 T248 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"customcollreplicadeletion_a_replica1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:55432/kwy",
   [junit4]   2> 	  "node_name":"127.0.0.1:55432_kwy",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"a",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 74998 T248 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 74999 T286 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 74999 T256 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 74999 T286 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 5 
   [junit4]   2> 75003 T256 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 5 
   [junit4]   2> 75647 T278 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for customcollreplicadeletion_b_replica1
   [junit4]   2> 75648 T278 oasc.ZkController.createCollectionZkNode Check for collection zkNode:customcollreplicadeletion
   [junit4]   2> 75648 T278 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 75649 T278 oascc.ZkStateReader.readConfigName Load collection config from:/collections/customcollreplicadeletion
   [junit4]   2> 75650 T278 oascc.ZkStateReader.readConfigName path=/collections/customcollreplicadeletion configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 75650 T278 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-DF1F884DADB46034-001/tempDir-004/customcollreplicadeletion_b_replica1/'
   [junit4]   2> 75686 T278 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 75733 T278 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 75734 T278 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 75741 T278 oass.IndexSchema.readSchema [customcollreplicadeletion_b_replica1] Schema name=test
   [junit4]   2> 75890 T278 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 75896 T278 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 75897 T278 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 75927 T278 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 75943 T278 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 75949 T278 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 75956 T278 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 75957 T278 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 75957 T278 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 75958 T278 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 75959 T278 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 75959 T278 oasc.CoreContainer.create Creating SolrCore 'customcollreplicadeletion_b_replica1' using configuration from collection customcollreplicadeletion
   [junit4]   2> 75959 T278 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 75960 T278 oasc.SolrCore.<init> [customcollreplicadeletion_b_replica1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-DF1F884DADB46034-001/tempDir-004/customcollreplicadeletion_b_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-DF1F884DADB46034-001/tempDir-004/customcollreplicadeletion_b_replica1/data/
   [junit4]   2> 75961 T278 oasc.JmxMonitoredMap.<init> JMX monitoring

[...truncated too long message...]

ctoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #10: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=6938, name=Thread-2500, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path11/7/17" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DF1F884DADB46034]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #11: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=6943, name=Thread-2505, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path11/7/17" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DF1F884DADB46034]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:884)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.UnixPath.checkWrite(UnixPath.java:801)
   [junit4]    > 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:376)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:674)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:727)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 2403804 T6934 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{}, sim=DefaultSimilarity, locale=ar_EG, timezone=Australia/West
   [junit4]   2> NOTE: Linux 3.13.0-35-generic amd64/Oracle Corporation 1.8.0_40-ea (64-bit)/cpus=8,threads=1,free=322777160,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [ExpressionTest, HdfsDirectoryTest, TestSchemaVersionResource, XsltUpdateRequestHandlerTest, TestChildDocTransformer, ModifyConfFileTest, TestReRankQParserPlugin, SolrTestCaseJ4Test, TestParamSet, TestFreeTextSuggestions, TestComponentsName, ExternalCollectionsTest, DeleteLastCustomShardedReplicaTest, CoreMergeIndexesAdminHandlerTest, IndexSchemaTest, CursorMarkTest, RollingRestartTest, TestLuceneMatchVersion, CurrencyFieldOpenExchangeTest, TestMiniSolrCloudCluster, TestSort, SyncSliceTest, TestSimpleQParserPlugin, FieldMutatingUpdateProcessorTest, BasicFunctionalityTest, TestCodecSupport, JSONWriterTest, TestCopyFieldCollectionResource, FastVectorHighlighterTest, TestSolr4Spatial, TestHashPartitioner, DocumentBuilderTest, TestReplicationHandler, AsyncMigrateRouteKeyTest, PingRequestHandlerTest, SpellingQueryConverterTest, TestSystemIdResolver, JsonLoaderTest, DistribDocExpirationUpdateProcessorTest, TestRecoveryHdfs, ResponseLogComponentTest, TestManagedStopFilterFactory, SharedFSAutoReplicaFailoverUtilsTest, TestShardHandlerFactory, TestConfig, TestLRUCache, DeleteShardTest, SolrCoreCheckLockOnStartupTest, TestFastLRUCache, TestDefaultSimilarityFactory, TestWordDelimiterFilterFactory, TestFieldResource, ZkCLITest, IndexBasedSpellCheckerTest, HighlighterConfigTest, TestSolrCoreProperties, StressHdfsTest, TestSolrJ, TestMissingGroups, LeaderElectionTest, SignatureUpdateProcessorFactoryTest, DirectUpdateHandlerTest, TestSolrXmlPersistence, OverseerCollectionProcessorTest, HdfsSyncSliceTest, SystemInfoHandlerTest, ReturnFieldsTest, AssignTest, TestStressUserVersions, HdfsRecoveryZkTest, TimeZoneUtilsTest, TestAnalyzeInfixSuggestions, TestLMDirichletSimilarityFactory, ZkNodePropsTest, StatelessScriptUpdateProcessorFactoryTest, MBeansHandlerTest, DocValuesMultiTest, TriLevelCompositeIdRoutingTest, PathHierarchyTokenizerFactoryTest, DocValuesMissingTest, TestDFRSimilarityFactory, SuggesterWFSTTest, TestCoreDiscovery, TestOrdValues, TestMaxScoreQueryParser, DistributedFacetPivotLargeTest, UnloadDistributedZkTest, PreAnalyzedUpdateProcessorTest, TestLMJelinekMercerSimilarityFactory, TestManagedResource, TestShortCircuitedRequests, DistributedExpandComponentTest, HighlighterMaxOffsetTest, TestRTGBase, TestSolrDeletionPolicy2, TestDocSet, QueryEqualityTest, RegexBoostProcessorTest, EchoParamsTest, TestManagedSynonymFilterFactory, DistributedQueryComponentCustomSortTest, DisMaxRequestHandlerTest, DistributedDebugComponentTest, SimpleFacetsTest, BasicDistributedZkTest, OverseerTest, LeaderElectionIntegrationTest, RecoveryZkTest, FullSolrCloudDistribCmdsTest, TestZkChroot, TestRandomDVFaceting, TestDistributedGrouping, ZkControllerTest, TestStressReorder, TestJoin, HardAutoCommitTest, TestRangeQuery, TestCoreContainer, StatsComponentTest, SpellCheckComponentTest, SolrCmdDistributorTest, BadIndexSchemaTest, TestFunctionQuery, TestBadConfig, HighlighterTest, ShowFileRequestHandlerTest, SolrIndexSplitterTest, SimplePostToolTest, AnalysisAfterCoreReloadTest, TestExtendedDismaxParser, SpellCheckCollatorTest, CoreAdminHandlerTest, TestTrie, WordBreakSolrSpellCheckerTest, TestUpdate, TestAtomicUpdateErrorCases, DirectUpdateHandlerOptimizeTest, SortByFunctionTest, TestRemoteStreaming, DirectSolrSpellCheckerTest, DocumentAnalysisRequestHandlerTest, TestQueryTypes, FileBasedSpellCheckerTest, RequestHandlersTest, TestValueSourceCache, TestIndexingPerformance, TestSolrQueryParser, TestJmxIntegration, TestAnalyzedSuggestions, BinaryUpdateRequestHandlerTest, CoreContainerCoreInitFailuresTest, SOLR749Test, UpdateParamsTest, TestQuerySenderNoQuery, BadComponentTest, SampleTest, TestSearchPerf, MinimalSchemaTest, TestFuzzyAnalyzedSuggestions, ExternalFileFieldSortTest, TestPostingsSolrHighlighter, DirectSolrConnectionTest, NotRequiredUniqueKeyTest, TestPhraseSuggestions, SpellPossibilityIteratorTest, TestXIncludeConfig, TestSweetSpotSimilarityFactory, TestPerFieldSimilarity, TestIBSimilarityFactory, TestBM25SimilarityFactory, TestFastWriter, LegacyHTMLStripCharFilterTest, TestJmxMonitoredMap, TestFastOutputStream, OpenExchangeRatesOrgProviderTest, URLClassifyProcessorTest, PluginInfoTest, ChaosMonkeyNothingIsSafeTest, TestHighlightDedupGrouping, FieldFacetTest, CollectionsAPIAsyncDistributedZkTest, DeleteReplicaTest, RemoteQueryErrorTest, TestLeaderElectionZkExpiry, TestModifyConfFiles, VMParamsZkACLAndCredentialsProvidersTest, HdfsBasicDistributedZk2Test, HdfsBasicDistributedZkTest, HdfsChaosMonkeySafeLeaderTest, HdfsCollectionsAPIDistributedZkTest, HdfsUnloadDistributedZkTest, CachingDirectoryFactoryTest]
   [junit4] Completed on J0 in 8.17s, 1 test, 1 error <<< FAILURES!

[...truncated 185 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:471: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:60: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1358: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:961: There were test failures: 438 suites, 1830 tests, 3 suite-level errors, 4 errors, 45 ignored (23 assumptions)

Total time: 90 minutes 40 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_40-ea-b04 -XX:+UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.9.0-ea-b28) - Build # 11239 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11239/
Java: 64bit/jdk1.9.0-ea-b28 -XX:-UseCompressedOops -XX:+UseParallelGC

All tests passed

Build Log:
[...truncated 50578 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:430: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:105: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:204: The following files are missing svn:eol-style (or binary svn:mime-type):
* ./dev-tools/idea/lucene/backward-codecs/backward-codecs.iml

Total time: 99 minutes 10 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.9.0-ea-b28 -XX:-UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.9.0-ea-b28) - Build # 11238 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11238/
Java: 32bit/jdk1.9.0-ea-b28 -client -XX:+UseSerialGC

All tests passed

Build Log:
[...truncated 31784 lines...]
-check-forbidden-all:
[forbidden-apis] Reading bundled API signatures: jdk-unsafe-1.7
[forbidden-apis] Reading bundled API signatures: jdk-deprecated-1.7
[forbidden-apis] Reading bundled API signatures: commons-io-unsafe-2.3
[forbidden-apis] Reading API signatures: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/tools/forbiddenApis/base.txt
[forbidden-apis] Reading API signatures: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/tools/forbiddenApis/servlet-api.txt
[forbidden-apis] Loading classes to check...
[forbidden-apis] Scanning for API signatures and dependencies...
[forbidden-apis] Forbidden method invocation: java.lang.String#<init>(byte[]) [Uses default charset]
[forbidden-apis]   in org.apache.solr.cloud.Overseer$ClusterStateUpdater (Overseer.java:316)
[forbidden-apis] Scanned 2357 (and 1560 related) class file(s) for forbidden API invocations (in 1.54s), 1 error(s).

BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:85: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:271: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:479: Check for forbidden API calls failed, see log.

Total time: 109 minutes 6 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.9.0-ea-b28 -client -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.9.0-ea-b28) - Build # 11237 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11237/
Java: 64bit/jdk1.9.0-ea-b28 -XX:+UseCompressedOops -XX:+UseG1GC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.testDistribSearch

Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:59445, http://127.0.0.1:41238, http://127.0.0.1:49769]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:59445, http://127.0.0.1:41238, http://127.0.0.1:49769]
	at __randomizedtesting.SeedInfo.seed([A4B8794CEC9E772E:255EF7549BC11712]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:322)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:874)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:658)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601)
	at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.removeAndWaitForLastReplicaGone(DeleteLastCustomShardedReplicaTest.java:116)
	at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.doTest(DeleteLastCustomShardedReplicaTest.java:106)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:484)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not  remove replica : customcollreplicadeletion/a/core_node1
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:550)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:343)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:304)
	... 46 more




Build Log:
[...truncated 11074 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/init-core-data-001
   [junit4]   2> 605298 T1826 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
   [junit4]   2> 605299 T1826 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 605301 T1826 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 605301 T1826 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 605302 T1827 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 605402 T1826 oasc.ZkTestServer.run start zk server on port:60927
   [junit4]   2> 605402 T1826 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 605403 T1826 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 605405 T1833 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a2c9dc7 name:ZooKeeperConnection Watcher:127.0.0.1:60927 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 605405 T1826 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 605406 T1826 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 605406 T1826 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 605408 T1826 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 605409 T1826 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 605410 T1835 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6d53930d name:ZooKeeperConnection Watcher:127.0.0.1:60927/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 605410 T1826 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 605410 T1826 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 605411 T1826 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 605412 T1826 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 605414 T1826 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 605415 T1826 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 605416 T1826 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 605416 T1826 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 605418 T1826 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 605419 T1826 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 605420 T1826 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 605421 T1826 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 605422 T1826 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 605422 T1826 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 605424 T1826 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 605424 T1826 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 605425 T1826 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 605425 T1826 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 605427 T1826 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 605427 T1826 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 605429 T1826 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 605429 T1826 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 605430 T1826 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 605430 T1826 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 605432 T1826 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 605432 T1826 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 605433 T1826 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 605433 T1826 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 605675 T1826 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 605678 T1826 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49769
   [junit4]   2> 605679 T1826 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 605679 T1826 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 605679 T1826 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-002
   [junit4]   2> 605680 T1826 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-002/'
   [junit4]   2> 605695 T1826 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-002/solr.xml
   [junit4]   2> 605704 T1826 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-002
   [junit4]   2> 605704 T1826 oasc.CoreContainer.<init> New CoreContainer 802725686
   [junit4]   2> 605704 T1826 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-002/]
   [junit4]   2> 605705 T1826 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 605705 T1826 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 605705 T1826 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 605705 T1826 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 605706 T1826 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 605706 T1826 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 605706 T1826 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 605706 T1826 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 605707 T1826 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 605707 T1826 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 605707 T1826 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 605708 T1826 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 605708 T1826 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 605708 T1826 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:60927/solr
   [junit4]   2> 605709 T1826 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 605709 T1826 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 605713 T1826 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 605714 T1846 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3b2399ec name:ZooKeeperConnection Watcher:127.0.0.1:60927 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 605714 T1826 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 605715 T1826 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 605716 T1826 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 605717 T1848 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@676b116e name:ZooKeeperConnection Watcher:127.0.0.1:60927/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 605717 T1826 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 605719 T1826 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 605721 T1826 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 605723 T1826 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 605725 T1826 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 605727 T1826 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 605728 T1826 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 605729 T1826 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49769_
   [junit4]   2> 605730 T1826 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49769_
   [junit4]   2> 605732 T1826 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 605733 T1826 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 605734 T1826 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 605735 T1826 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:49769_
   [junit4]   2> 605735 T1826 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 605736 T1826 oasc.Overseer.start Overseer (id=92445915764817923-127.0.0.1:49769_-n_0000000000) starting
   [junit4]   2> 605737 T1826 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 605740 T1826 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 605741 T1850 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 605741 T1826 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 605742 T1826 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 605743 T1826 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 605745 T1849 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 605745 T1826 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-002
   [junit4]   2> 605753 T1826 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-002/conf/
   [junit4]   2> 605870 T1826 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 605871 T1826 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 605871 T1826 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 605871 T1826 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 605872 T1826 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 605873 T1854 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@54c3f3c9 name:ZooKeeperConnection Watcher:127.0.0.1:60927/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 605873 T1826 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 605873 T1826 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 605874 T1826 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 605876 T1826 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 605876 T1826 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 605876 T1826 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 605876 T1826 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 605877 T1856 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3f89e748 name:ZooKeeperConnection Watcher:127.0.0.1:60927/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 605877 T1826 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 605877 T1826 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 605879 T1857 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 605883 T1849 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: collection1
   [junit4]   2> 605883 T1849 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 605884 T1849 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 2
   [junit4]   2> 605885 T1849 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/collection1/state.json
   [junit4]   2> 605885 T1857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 605885 T1858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 606148 T1826 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 606149 T1826 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 606151 T1826 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59445
   [junit4]   2> 606151 T1826 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 606152 T1826 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 606152 T1826 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-003
   [junit4]   2> 606152 T1826 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-003/'
   [junit4]   2> 606161 T1826 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-003/solr.xml
   [junit4]   2> 606167 T1826 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-003
   [junit4]   2> 606167 T1826 oasc.CoreContainer.<init> New CoreContainer 989850544
   [junit4]   2> 606167 T1826 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-003/]
   [junit4]   2> 606168 T1826 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 606168 T1826 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 606168 T1826 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 606168 T1826 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 606168 T1826 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 606169 T1826 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 606169 T1826 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 606169 T1826 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 606169 T1826 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 606170 T1826 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 606170 T1826 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 606170 T1826 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 606171 T1826 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 606171 T1826 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:60927/solr
   [junit4]   2> 606171 T1826 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 606171 T1826 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 606172 T1826 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 606173 T1869 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@34447502 name:ZooKeeperConnection Watcher:127.0.0.1:60927 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 606174 T1826 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 606174 T1826 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 606175 T1826 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 606176 T1871 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@44d8cbf8 name:ZooKeeperConnection Watcher:127.0.0.1:60927/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 606176 T1826 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 606181 T1826 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 607184 T1826 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59445_
   [junit4]   2> 607186 T1826 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59445_
   [junit4]   2> 607190 T1826 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 607193 T1826 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-003
   [junit4]   2> 607204 T1826 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-003/conf/
   [junit4]   2> 607318 T1826 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 607319 T1826 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 607320 T1826 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 607548 T1826 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 607549 T1826 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 607551 T1826 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:41238
   [junit4]   2> 607552 T1826 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 607552 T1826 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 607553 T1826 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-004
   [junit4]   2> 607553 T1826 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-004/'
   [junit4]   2> 607568 T1826 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-004/solr.xml
   [junit4]   2> 607573 T1826 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-004
   [junit4]   2> 607573 T1826 oasc.CoreContainer.<init> New CoreContainer 372160997
   [junit4]   2> 607574 T1826 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-004/]
   [junit4]   2> 607574 T1826 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 607574 T1826 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 607574 T1826 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 607575 T1826 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 607575 T1826 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 607575 T1826 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 607575 T1826 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 607575 T1826 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 607576 T1826 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 607576 T1826 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 607576 T1826 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 607576 T1826 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 607577 T1826 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 607577 T1826 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:60927/solr
   [junit4]   2> 607577 T1826 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 607577 T1826 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 607578 T1826 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 607579 T1884 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4014c58a name:ZooKeeperConnection Watcher:127.0.0.1:60927 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 607580 T1826 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 607581 T1826 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 607582 T1826 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 607583 T1886 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5c83b5b8 name:ZooKeeperConnection Watcher:127.0.0.1:60927/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 607584 T1826 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 607588 T1826 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 608591 T1826 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41238_
   [junit4]   2> 608592 T1826 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41238_
   [junit4]   2> 608597 T1826 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 608600 T1826 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-004
   [junit4]   2> 608612 T1826 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-004/conf/
   [junit4]   2> 608755 T1826 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 608756 T1826 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 608756 T1826 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 608758 T1826 oasc.AbstractFullDistribZkTestBase.createCollection Creating collection with stateFormat=2: customcollreplicadeletion
   [junit4]   2> 608758 T1826 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 608759 T1826 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 608760 T1890 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3f6c1de6 name:ZooKeeperConnection Watcher:127.0.0.1:60927/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 608760 T1826 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 608760 T1826 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 608761 T1826 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 608768 T1866 oasha.CollectionsHandler.handleCreateAction Creating Collection : replicationFactor=1&shards=a,b&maxShardsPerNode=5&router.name=implicit&name=customcollreplicadeletion&action=CREATE&numShards=1&stateFormat=2&wt=javabin&version=2
   [junit4]   2> 608770 T1857 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 608771 T1850 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"customcollreplicadeletion",
   [junit4]   2> 	  "replicationFactor":"1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "maxShardsPerNode":"5",
   [junit4]   2> 	  "shards":"a,b",
   [junit4]   2> 	  "stateFormat":"2",
   [junit4]   2> 	  "router.name":"implicit"}
   [junit4]   2> 608771 T1891 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"customcollreplicadeletion",
   [junit4]   2> 	  "replicationFactor":"1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "maxShardsPerNode":"5",
   [junit4]   2> 	  "shards":"a,b",
   [junit4]   2> 	  "stateFormat":"2",
   [junit4]   2> 	  "router.name":"implicit"}
   [junit4]   2> 608773 T1891 oasc.OverseerCollectionProcessor.createConfNode Only one config set found in zk - using it:conf1
   [junit4]   2> 608773 T1891 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/customcollreplicadeletion 
   [junit4]   2> 608774 T1891 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion
   [junit4]   2> 608777 T1857 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 608780 T1849 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: customcollreplicadeletion
   [junit4]   2> 608780 T1849 oasc.Overseer$ClusterStateUpdater.createCollection Create collection customcollreplicadeletion with shards [a, b]
   [junit4]   2> 608781 T1849 oasc.Overseer$ClusterStateUpdater.createCollection state version customcollreplicadeletion 2
   [junit4]   2> 608783 T1849 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 608784 T1872 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 608784 T1858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 608784 T1857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 608784 T1892 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 608785 T1887 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 608877 T1891 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection customcollreplicadeletion, shardNames [a, b] , replicationFactor : 1
   [junit4]   2> 608878 T1891 oasc.OverseerCollectionProcessor.createCollection Creating shard customcollreplicadeletion_a_replica1 as part of slice a of collection customcollreplicadeletion on 127.0.0.1:49769_
   [junit4]   2> 608878 T1891 oasc.OverseerCollectionProcessor.createCollection Creating shard customcollreplicadeletion_b_replica1 as part of slice b of collection customcollreplicadeletion on 127.0.0.1:41238_
   [junit4]   2> 608881 T1842 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_a_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=a&wt=javabin&version=2
   [junit4]   2> 608881 T1882 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_b_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=b&wt=javabin&version=2
   [junit4]   2> 608883 T1842 oasc.ZkController.publish publishing core=customcollreplicadeletion_a_replica1 state=down collection=customcollreplicadeletion
   [junit4]   2> 608883 T1882 oasc.ZkController.publish publishing core=customcollreplicadeletion_b_replica1 state=down collection=customcollreplicadeletion
   [junit4]   2> 608884 T1857 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 608884 T1842 oasc.ZkController.preRegister Registering watch for external collection customcollreplicadeletion
   [junit4]   2> 608885 T1882 oasc.ZkController.preRegister Registering watch for external collection customcollreplicadeletion
   [junit4]   2> 608885 T1882 oascc.ZkStateReader.addZkWatch addZkWatch customcollreplicadeletion
   [junit4]   2> 608885 T1882 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/customcollreplicadeletion/state.json from ZooKeeper... 
   [junit4]   2> 608886 T1849 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"customcollreplicadeletion_a_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:49769",
   [junit4]   2> 	  "node_name":"127.0.0.1:49769_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"a",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 608886 T1882 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 0 
   [junit4]   2> 608887 T1882 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 608891 T1849 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 608891 T1887 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 608892 T1887 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 1 
   [junit4]   2> 608893 T1842 oascc.ZkStateReader.addZkWatch addZkWatch customcollreplicadeletion
   [junit4]   2> 608893 T1842 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/customcollreplicadeletion/state.json from ZooKeeper... 
   [junit4]   2> 608894 T1842 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 1 
   [junit4]   2> 608895 T1842 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 608895 T1842 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for customcollreplicadeletion_a_replica1
   [junit4]   2> 608895 T1842 oasc.ZkController.createCollectionZkNode Check for collection zkNode:customcollreplicadeletion
   [junit4]   2> 608895 T1842 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 608896 T1842 oascc.ZkStateReader.readConfigName Load collection config from:/collections/customcollreplicadeletion
   [junit4]   2> 608896 T1842 oascc.ZkStateReader.readConfigName path=/collections/customcollreplicadeletion configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 608896 T1842 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-002/customcollreplicadeletion_a_replica1/'
   [junit4]   2> 608898 T1849 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 1 
   [junit4]   2> 608898 T1849 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"customcollreplicadeletion_b_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:41238",
   [junit4]   2> 	  "node_name":"127.0.0.1:41238_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"b",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 608900 T1857 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 608901 T1849 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 608901 T1887 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 608901 T1857 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 608902 T1887 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 2 
   [junit4]   2> 608903 T1857 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 2 
   [junit4]   2> 608920 T1842 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 608954 T1842 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 608955 T1842 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 608960 T1842 oass.IndexSchema.readSchema [customcollreplicadeletion_a_replica1] Schema name=test
   [junit4]   2> 609047 T1842 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 609051 T1842 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 609053 T1842 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 609060 T1842 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 609063 T1842 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 609065 T1842 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 609066 T1842 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 609067 T1842 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 609068 T1842 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 609068 T1842 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 609069 T1842 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 609069 T1842 oasc.CoreContainer.create Creating SolrCore 'customcollreplicadeletion_a_replica1' using configuration from collection customcollreplicadeletion
   [junit4]   2> 609069 T1842 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 609069 T1842 oasc.SolrCore.<init> [customcollreplicadeletion_a_replica1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-002/customcollreplicadeletion_a_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-002/customcollreplicadeletion_a_replica1/data/
   [junit4]   2> 609070 T1842 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2daaa4c0
   [junit4]   2> 609070 T1842 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-002/customcollreplicadeletion_a_replica1/data
   [junit4]   2> 609071 T1842 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-002/customcollreplicadeletion_a_replica1/data/index/
   [junit4]   2> 609071 T1842 oasc.SolrCore.initIndex WARN [customcollreplicadeletion_a_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-002/customcollreplicadeletion_a_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 609072 T1842 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-002/customcollreplicadeletion_a_replica1/data/index
   [junit4]   2> 609072 T1842 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=26, maxMergedSegmentMB=58.4697265625, floorSegmentMB=1.232421875, forceMergeDeletesPctAllowed=23.442905684858, segmentsPerTier=29.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 609073 T1842 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@6d6fa5d9 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@fbbd395),segFN=segments_1,generation=1}
   [junit4]   2> 609073 T1842 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 609077 T1842 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 609077 T1842 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 609078 T1842 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 609078 T1842 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 609078 T1842 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 609079 T1842 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 609079 T1842 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 609079 T1842 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 609080 T1842 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 609080 T1842 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 609081 T1842 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 609081 T1842 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 609082 T1842 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 609082 T1842 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 609082 T1842 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 609083 T1842 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 609083 T1842 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 609083 T1842 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 609084 T1842 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 609084 T1842 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 609087 T1842 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 609088 T1842 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 609090 T1842 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 609091 T1842 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 609093 T1842 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 609093 T1842 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 609094 T1842 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=15, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 609094 T1842 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@6d6fa5d9 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@fbbd395),segFN=segments_1,generation=1}
   [junit4]   2> 609095 T1842 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 609095 T1842 oass.SolrIndexSearcher.<init> Opening Searcher@59fc3c28[customcollreplicadeletion_a_replica1] main
   [junit4]   2> 609095 T1842 oascc.ZkStateReader.readConfigName Load collection config from:/collections/customcollreplicadeletion
   [junit4]   2> 609096 T1842 oascc.ZkStateReader.readConfigName path=/collections/customcollreplicadeletion configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 609096 T1842 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 609097 T1842 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 609097 T1842 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 609097 T1842 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 609098 T1842 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 609098 T1842 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 609099 T1842 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 609100 T1842 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 609100 T1842 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 609101 T1842 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 609102 T1895 oasc.SolrCore.registerSearcher [customcollreplicadeletion_a_replica1] Registered new searcher Searcher@59fc3c28[customcollreplicadeletion_a_replica1] main{UninvertingDirectoryReader()}
   [junit4]   2> 609105 T1842 oasc.CoreContainer.registerCore registering core: customcollreplicadeletion_a_replica1
   [junit4]   2> 609105 T1842 oasc.ZkController.register Register replica - core:customcollreplicadeletion_a_replica1 address:http://127.0.0.1:49769 collection:customcollreplicadeletion shard:a
   [junit4]   2> 609106 T1842 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion/leader_elect/a/election
   [junit4]   2> 609109 T1842 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard a
   [junit4]   2> 609110 T1857 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 609110 T1842 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 609110 T1842 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C318 name=customcollreplicadeletion_a_replica1 org.apache.solr.core.SolrCore@168cf0a4 url=http://127.0.0.1:49769/customcollreplicadeletion_a_replica1 node=127.0.0.1:49769_ C318_STATE=coll:customcollreplicadeletion core:customcollreplicadeletion_a_replica1 props:{core=customcollreplicadeletion_a_replica1, base_url=http://127.0.0.1:49769, node_name=127.0.0.1:49769_, state=down}
   [junit4]   2> 609111 T1842 C318 P49769 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:49769/customcollreplicadeletion_a_replica1/
   [junit4]   2> 609111 T1842 C318 P49769 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 609112 T1842 C318 P49769 oasc.SyncStrategy.syncToMe http://127.0.0.1:49769/customcollreplicadeletion_a_replica1/ has no replicas
   [junit4]   2> 609112 T1842 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:49769/customcollreplicadeletion_a_replica1/ a
   [junit4]   2> 609112 T1842 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion/leaders/a
   [junit4]   2> 609113 T1849 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 2 
   [junit4]   2> 609114 T1849 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 609115 T1857 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 609115 T1887 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 609116 T1887 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 3 
   [junit4]   2> 609116 T1857 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 3 
   [junit4]   2> 609119 T1849 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 3 
   [junit4]   2> 609120 T1857 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 609120 T1849 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 609120 T1857 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 609121 T1887 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 609121 T1887 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 4 
   [junit4]   2> 609122 T1857 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 4 
   [junit4]   2> 609167 T1842 oasc.ZkController.register We are http://127.0.0.1:49769/customcollreplicadeletion_a_replica1/ and leader is http://127.0.0.1:49769/customcollreplicadeletion_a_replica1/
   [junit4]   2> 609167 T1842 oasc.ZkController.register No LogReplay needed for core=customcollreplicadeletion_a_replica1 baseURL=http://127.0.0.1:49769
   [junit4]   2> 609167 T1842 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 609167 T1842 oasc.ZkController.publish publishing core=customcollreplicadeletion_a_replica1 state=active collection=customcollreplicadeletion
   [junit4]   2> 609168 T1857 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 609170 T1842 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 4 
   [junit4]   2> 609171 T1842 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_a_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=a&wt=javabin&version=2} status=0 QTime=290 
   [junit4]   2> 609172 T1849 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 4 
   [junit4]   2> 609173 T1849 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"customcollreplicadeletion_a_replica1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:49769",
   [junit4]   2> 	  "node_name":"127.0.0.1:49769_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"a",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 609174 T1849 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 609175 T1887 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 609175 T1857 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 609176 T1887 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 5 
   [junit4]   2> 609177 T1857 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 5 
   [junit4]   2> 609887 T1882 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for customcollreplicadeletion_b_replica1
   [junit4]   2> 609887 T1882 oasc.ZkController.createCollectionZkNode Check for collection zkNode:customcollreplicadeletion
   [junit4]   2> 609888 T1882 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 609888 T1882 oascc.ZkStateReader.readConfigName Load collection config from:/collections/customcollreplicadeletion
   [junit4]   2> 609889 T1882 oascc.ZkStateReader.readConfigName path=/collections/customcollreplicadeletion configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 609889 T1882 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-004/customcollreplicadeletion_b_replica1/'
   [junit4]   2> 609907 T1882 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 609916 T1882 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 609917 T1882 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 609922 T1882 oass.IndexSchema.readSchema [customcollreplicadeletion_b_replica1] Schema name=test
   [junit4]   2> 610014 T1882 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 610017 T1882 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 610019 T1882 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 610025 T1882 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 610027 T1882 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 610028 T1882 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 610029 T1882 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 610029 T1882 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 610029 T1882 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 610029 T1882 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 610030 T1882 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 610030 T1882 oasc.CoreContainer.create Creating SolrCore 'customcollreplicadeletion_b_replica1' using configuration from collection customcollreplicadeletion
   [junit4]   2> 610030 T1882 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 610030 T1882 oasc.SolrCore.<init> [customcollreplicadeletion_b_replica1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-004/customcollreplicadeletion_b_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-004/customcollreplicadeletion_b_replica1/data/
   [junit4]   2> 610031 T1882 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2daaa4c0
   [junit4]   2> 610031 T1882 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-004/customcollreplicadeletion_b_replica1/data
   [junit4]   2> 610032 T1882 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-004/customcollreplicadeletion_b_replica1/data/index/
   [junit4]   2> 610032 T1882 oasc.SolrCore.initIndex WARN [customcollreplicadeletion_b_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-004/customcollreplicadeletion_b_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 610032 T1882 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-004/customcollreplicadeletion_b_replica1/data/index
   [junit4]   2> 610032 T1882 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=26, maxMergedSegmentMB=58.4697265625, floorSegmentMB=1.232421875, forceMergeDeletesPctAllowed=23.442905684858, segmentsPerTier=29.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 610033 T1882 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@7f909c1e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4f873323),segFN=segments_1,generation=1}
   [junit4]   2> 610033 T1882 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 610036 T1882 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 610036 T1882 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 610036 T1882 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 610036 T1882 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 610036 T1882 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 610037 T1882 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 610037 T1882 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 610037 T1882 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 610037 T1882 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 610038 T1882 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 610038 T1882 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 610038 T1882 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 610039 T1882 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 610039 T1882 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 610039 T1882 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 610039 T1882 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 610040 T1882 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 610040 T1882 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 610040 T1882 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 610040 T1882 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 610041 T1882 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 610042 T1882 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 610043 T1882 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 610043 T1882 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 610045 T1882 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 610045 T1882 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 610045 T1882 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=15, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 610046 T1882 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@7f909c1e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4f873323),segFN=segments_1,generation=1}
   [junit4]   2> 610046 T1882 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 610046 T1882 oass.SolrIndexSearcher.<init> Opening Searcher@7da73d1[customcollreplicadeletion_b_replica1] main
   [junit4]   2> 610046 T1882 oascc.ZkStateReader.readConfigName Load collection config from:/collections/customcollreplicadeletion
   [junit4]   2> 610047 T1882 oascc.ZkStateReader.readConfigName path=/collections/customcollreplicadeletion configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 610047 T1882 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 610048 T1882 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 610048 T1882 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 610048 T1882 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 610049 T1882 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 610049 T1882 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 610049 T1882 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 610049 T1882 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 610051 T1897 oasc.SolrCore.registerSearcher [customcollreplicadeletion_b_replica1] Registered new searcher Searcher@7da73d1[customcollreplicadeletion_b_replica1] main{UninvertingDirectoryReader()}
   [junit4]   2> 610052 T1882 oasc.CoreContainer.registerCore registering core: customcollreplicadeletion_b_replica1
   [junit4]   2> 610053 T1882 oasc.ZkController.register Register replica - core:customcollreplicadeletion_b_replica1 address:http://127.0.0.1:41238 collection:customcollreplicadeletion shard:b
   [junit4]   2> 610053 T1882 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion/leader_elect/b/election
   [junit4]   2> 610056 T1882 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard b
   [junit4]   2> 610057 T1857 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 610057 T1882 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 610058 T1882 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C319 name=customcollreplicadeletion_b_replica1 org.apache.solr.core.SolrCore@7973bdb7 url=http://127.0.0.1:41238/customcollreplicadeletion_b_replica1 node=127.0.0.1:41238_ C319_STATE=coll:customcollreplicadeletion core:customcollreplicadeletion_b_replica1 props:{core=customcollreplicadeletion_b_replica1, base_url=http://127.0.0.1:41238, node_name=127.0.0.1:41238_, state=down}
   [junit4]   2> 610058 T1882 C319 P41238 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:41238/customcollreplicadeletion_b_replica1/
   [junit4]   2> 610058 T1882 C319 P41238 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 610058 T1882 C319 P41238 oasc.SyncStrategy.syncToMe http://127.0.0.1:41238/customcollreplicadeletion_b_replica1/ has no replicas
   [junit4]   2> 610058 T1882 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:41238/customcollreplicadeletion_b_replica1/ b
   [junit4]   2> 610059 T1882 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion/leaders/b
   [junit4]   2> 610059 T1849 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 5 
   [junit4]   2> 610061 T1849 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 610061 T1857 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 610062 T1887 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 610062 T1887 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 6 
   [junit4]   2> 610063 T1857 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 6 
   [junit4]   2> 610065 T1849 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 6 
   [junit4]   2> 610066 T1857 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 610066 T1849 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 610067 T1857 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 610067 T1887 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 610067 T1887 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 7 
   [junit4]   2> 610068 T1857 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 7 
   [junit4]   2> 610112 T1882 oasc.ZkController.register We are http://127.0.0.1:41238/customcollreplicadeletion_b_replica1/ and leader is http://127.0.0.1:41238/customcollreplicadeletion_b_replica1/
   [junit4]   2> 610112 T1882 oasc.ZkController.register No LogReplay needed for core=customcollreplicadeletion_b_replica1 baseURL=http://127.0.0.1:41238
   [junit4]   2> 610113 T1882 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 610113 T1882 oasc.ZkController.publish publishing core=customcollreplicadeletion_b_replica1 state=active collection=customcollreplicadeletion
   [junit4]   2> 610114 T1857 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 610116 T1882 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 7 
   [junit4]   2> 610116 T1849 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 7 
   [junit4]   2> 610116 T1882 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_b_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=b&wt=javabin&version=2} status=0 QTime=1235 
   [junit4]   2> 610117 T1849 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"customcollreplicadeletion_b_replica1",
   [junit4]   2> 	  "core_node_name":"core_node2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:41238",
   [junit4]   2> 	  "node_name":"127.0.0.1:41238_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"b",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 610117 T1891 oasc.OverseerCollectionProcessor.createCollection Finished create command on all shards for collection: customcollreplicadeletion
   [junit4]   2> 610118 T1849 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 610119 T1857 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 610119 T1887 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 610119 T1887 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 8 
   [junit4]   2> 610120 T1891 oasc.OverseerCollectionProcessor$Runner.run Overseer Collection Processor: Message id:/overseer/collection-queue-work/qn-0000000000 complete, response:{success={null={responseHeader={status=0,QTime=290},core=customcollreplicadeletion_a_replica1},null={responseHeader={status=0,QTime=1235},core=customcollreplicadeletion_b_replica1}}}
   [junit4]   2> 610120 T1857 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 8 
   [junit4]   2> 610121 T1872 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work/qnr-0000000000 state: SyncConnected type NodeDataChanged
   [junit4]   2> 610121 T1857 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 610122 T1866 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={replicationFactor=1&shards=a,b&maxShardsPerNode=5&router.name=implicit&name=customcollreplicadeletion&action=CREATE&numShards=1&stateFormat=2&wt=javabin&version=2} status=0 QTime=1354 
   [junit4]   2> 610122 T1826 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: customcollreplicadeletion failOnTimeout:true timeout (sec):330
   [junit4]   2> 610124 T1826 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: customcollreplicadeletion
   [junit4]   2> 610124 T1826 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 610125 T1826 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 610126 T1900 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3ab0ccfd name:ZooKeeperConnection Watcher:127.0.0.1:60927/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 610126 T1826 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 610126 T1826 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 610127 T1826 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 610132 T1864 oasha.CollectionsHandler.handleRemoveReplica Remove replica: replica=core_node1&action=deletereplica&collection=customcollreplicadeletion&shard=a&wt=javabin&version=2
   [junit4]   2> 610133 T1857 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 610134 T1850 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000002 message:{
   [junit4]   2> 	  "operation":"deletereplica",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "shard":"a",
   [junit4]   2> 	  "replica":"core_node1"}
   [junit4]   2> 610134 T1901 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : deletereplica , {
   [junit4]   2> 	  "operation":"deletereplica",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "shard":"a",
   [junit4]   2> 	  "replica":"core_node1"}
   [junit4]   2> 610137 T1843 oasc.SolrCore.close [customcollreplicadeletion_a_replica1]  CLOSING SolrCore org.apache.solr.core.SolrCore@168cf0a4
   [junit4]   2> 610145 T1843 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
   [junit4]   2> 610146 T1843 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 610146 T1843 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 610146 T1843 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 610148 T1843 oasc.SolrCore.closeSearcher [customcollreplicadeletion_a_replica1] Closing main searcher on request.
   [junit4]   2> 610149 T1843 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 610149 T1843 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-002/customcollreplicadeletion_a_replica1/data [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-002/customcollreplicadeletion_a_replica1/data;done=false>>]
   [junit4]   2> 610149 T1843 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-002/customcollreplicadeletion_a_replica1/data/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-002/customcollreplicadeletion_a_replica1/data/index;done=false>>, CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-002/customcollreplicadeletion_a_replica1/data;done=false>>]
   [junit4]   2> 610150 T1843 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-002/customcollreplicadeletion_a_replica1/data/index
   [junit4]   2> 610150 T1843 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-002/customcollreplicadeletion_a_replica1/data
   [junit4]   2> 610150 T1843 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-002/customcollreplicadeletion_a_replica1/data []
   [junit4]   

[...truncated too long message...]

actory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-004/customcollreplicadeletion_b_replica1/data/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-004/customcollreplicadeletion_b_replica1/data/index;done=false>>]
   [junit4]   2> 728365 T1826 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-004/customcollreplicadeletion_b_replica1/data/index
   [junit4]   2> 728365 T1826 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-004/customcollreplicadeletion_b_replica1/data [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-004/customcollreplicadeletion_b_replica1/data;done=false>>]
   [junit4]   2> 728366 T1826 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001/tempDir-004/customcollreplicadeletion_b_replica1/data
   [junit4]   2> 728366 T1826 oasc.Overseer.close Overseer (id=92445915764817929-127.0.0.1:41238_-n_0000000002) closing
   [junit4]   2> 728367 T1916 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:41238_
   [junit4]   2> 729867 T1919 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:347)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:344)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:344)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:346)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:296)
   [junit4]   2> 
   [junit4]   2> 729867 T1915 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 729898 T1826 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 734884 T1826 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 734887 T1826 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:60927 60927
   [junit4]   2> 735573 T1827 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:60927 60927
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DeleteLastCustomShardedReplicaTest -Dtests.method=testDistribSearch -Dtests.seed=A4B8794CEC9E772E -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=mt -Dtests.timezone=America/Detroit -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR    130s J0 | DeleteLastCustomShardedReplicaTest.testDistribSearch <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:59445, http://127.0.0.1:41238, http://127.0.0.1:49769]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A4B8794CEC9E772E:255EF7549BC11712]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:322)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:874)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:658)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601)
   [junit4]    > 	at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.removeAndWaitForLastReplicaGone(DeleteLastCustomShardedReplicaTest.java:116)
   [junit4]    > 	at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.doTest(DeleteLastCustomShardedReplicaTest.java:106)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not  remove replica : customcollreplicadeletion/a/core_node1
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:550)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:343)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:304)
   [junit4]    > 	... 46 more
   [junit4]   2> 735691 T1826 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-A4B8794CEC9E772E-001
   [junit4]   2> 130398 T1825 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=mt, timezone=America/Detroit
   [junit4]   2> NOTE: Linux 3.13.0-35-generic amd64/Oracle Corporation 1.9.0-ea (64-bit)/cpus=8,threads=1,free=192250224,total=454033408
   [junit4]   2> NOTE: All tests run in this JVM: [TestSolrXmlPersistence, HdfsChaosMonkeySafeLeaderTest, TestManagedStopFilterFactory, TestPartialUpdateDeduplication, UUIDUpdateProcessorFallbackTest, CursorPagingTest, TestFastLRUCache, TestLuceneMatchVersion, TestStressRecovery, ChaosMonkeyNothingIsSafeTest, OverseerTest, SpellPossibilityIteratorTest, TestChildDocTransformer, TestDefaultSimilarityFactory, TestReloadAndDeleteDocs, IndexSchemaTest, SpellCheckComponentTest, TestMissingGroups, TestSchemaResource, NoCacheHeaderTest, TestComponentsName, TestFaceting, ExpressionTest, PluginInfoTest, HdfsDirectoryTest, MBeansHandlerTest, DistributedSuggestComponentTest, FastVectorHighlighterTest, OverseerCollectionProcessorTest, TestSort, DistributedSpellCheckComponentTest, TestReload, ChangedSchemaMergeTest, TestTrie, TestFreeTextSuggestions, NoFacetTest, IndexBasedSpellCheckerTest, TestFieldResource, TestBinaryResponseWriter, TermsComponentTest, TriLevelCompositeIdRoutingTest, PathHierarchyTokenizerFactoryTest, TestHashPartitioner, RollingRestartTest, TestDistributedSearch, TimeZoneUtilsTest, TestManagedResource, PeerSyncTest, TestStressReorder, RegexBoostProcessorTest, DeleteLastCustomShardedReplicaTest]
   [junit4] Completed on J0 in 130.67s, 1 test, 1 error <<< FAILURES!

[...truncated 994 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:471: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:60: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1358: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:961: There were test failures: 438 suites, 1830 tests, 1 error, 45 ignored (23 assumptions)

Total time: 88 minutes 38 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.9.0-ea-b28 -XX:+UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (32bit/ibm-j9-jdk7) - Build # 11236 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11236/
Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}

1 tests failed.
REGRESSION:  org.apache.solr.cloud.SyncSliceTest.testDistribSearch

Error Message:
expected:<5> but was:<4>

Stack Trace:
java.lang.AssertionError: expected:<5> but was:<4>
	at __randomizedtesting.SeedInfo.seed([C9793CC7A981939A:489FB2DFDEDEF3A6]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:176)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:619)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:853)




Build Log:
[...truncated 10964 lines...]
   [junit4] Suite: org.apache.solr.cloud.SyncSliceTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/init-core-data-001
   [junit4]   2> 764265 T1793 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 764266 T1793 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /_nq/dk
   [junit4]   2> 764268 T1793 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 764268 T1793 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 764269 T1794 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 764369 T1793 oasc.ZkTestServer.run start zk server on port:35479
   [junit4]   2> 764370 T1793 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 764372 T1793 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 764377 T1800 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f1f6d55c name:ZooKeeperConnection Watcher:127.0.0.1:35479 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 764377 T1793 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 764378 T1793 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 764379 T1793 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 764383 T1793 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 764384 T1793 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 764386 T1802 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a2e004e name:ZooKeeperConnection Watcher:127.0.0.1:35479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 764387 T1793 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 764388 T1793 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 764389 T1793 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 764393 T1793 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 764396 T1793 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 764399 T1793 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 764402 T1793 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 764403 T1793 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 764408 T1793 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 764410 T1793 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 764412 T1793 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 764413 T1793 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 764415 T1793 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 764415 T1793 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 764417 T1793 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 764418 T1793 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 764420 T1793 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 764420 T1793 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 764423 T1793 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 764424 T1793 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 764427 T1793 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 764428 T1793 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 764430 T1793 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 764431 T1793 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 764434 T1793 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 764435 T1793 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 764437 T1793 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 764438 T1793 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 764696 T1793 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 764699 T1793 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:?????
   [junit4]   2> 764700 T1793 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 764701 T1793 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 764701 T1793 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-002
   [junit4]   2> 764702 T1793 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-002/'
   [junit4]   2> 764710 T1793 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-002/solr.xml
   [junit4]   2> 764743 T1793 oasc.CoreContainer.<init> New CoreContainer -1482570410
   [junit4]   2> 764744 T1793 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-002/]
   [junit4]   2> 764745 T1793 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 764745 T1793 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 764746 T1793 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 764746 T1793 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 764746 T1793 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 764747 T1793 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 764747 T1793 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 764748 T1793 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 764748 T1793 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 764749 T1793 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 764750 T1793 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 764750 T1793 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 764750 T1793 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 764751 T1793 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35479/solr
   [junit4]   2> 764751 T1793 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 764752 T1793 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 764753 T1793 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 764754 T1813 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@eb6bf433 name:ZooKeeperConnection Watcher:127.0.0.1:35479 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 764755 T1793 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 764755 T1793 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 764757 T1793 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 764758 T1815 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11527f75 name:ZooKeeperConnection Watcher:127.0.0.1:35479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 764758 T1793 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 764760 T1793 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 764765 T1793 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 764769 T1793 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 764774 T1793 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 764779 T1793 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 764783 T1793 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 764786 T1793 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48367__nq%2Fdk
   [junit4]   2> 764788 T1793 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48367__nq%2Fdk
   [junit4]   2> 764793 T1793 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 764796 T1793 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 764799 T1793 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 764802 T1793 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:48367__nq%2Fdk
   [junit4]   2> 764802 T1793 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 764805 T1793 oasc.Overseer.start Overseer (id=92445115034435587-127.0.0.1:48367__nq%2Fdk-n_0000000000) starting
   [junit4]   2> 764809 T1793 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 764818 T1793 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 764820 T1817 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 764820 T1793 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 764824 T1793 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 764826 T1793 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 764831 T1816 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 764836 T1819 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 764836 T1819 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 764838 T1819 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 764838 T1820 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 764844 T1816 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "node_name":"127.0.0.1:48367__nq%2Fdk",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48367/_nq/dk"}
   [junit4]   2> 764844 T1816 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 764845 T1816 oasc.Overseer$ClusterStateUpdater.createCollection state version control_collection 1
   [junit4]   2> 764845 T1816 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 764849 T1820 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 765839 T1819 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 765839 T1819 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 765841 T1819 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 765841 T1819 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 765843 T1819 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 765844 T1819 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-002/collection1/'
   [junit4]   2> 765845 T1819 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 765846 T1819 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 765846 T1819 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 765882 T1819 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 765917 T1819 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 765919 T1819 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 765924 T1819 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 766251 T1819 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 766257 T1819 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 766259 T1819 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 766274 T1819 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 766282 T1819 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 766288 T1819 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 766289 T1819 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 766290 T1819 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 766290 T1819 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 766291 T1819 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 766292 T1819 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 766292 T1819 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 766292 T1819 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 766293 T1819 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-001/control/data/
   [junit4]   2> 766293 T1819 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@80ed7c2e
   [junit4]   2> 766294 T1819 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-001/control/data
   [junit4]   2> 766294 T1819 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-001/control/data/index/
   [junit4]   2> 766295 T1819 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 766295 T1819 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-001/control/data/index
   [junit4]   2> 766296 T1819 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=27, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5827239501573246]
   [junit4]   2> 766297 T1819 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@a9a903a4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@996e68ab),segFN=segments_1,generation=1}
   [junit4]   2> 766297 T1819 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 766299 T1819 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 766300 T1819 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 766300 T1819 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 766301 T1819 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 766301 T1819 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 766301 T1819 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 766302 T1819 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 766302 T1819 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 766303 T1819 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 766303 T1819 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 766304 T1819 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 766304 T1819 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 766304 T1819 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 766305 T1819 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 766305 T1819 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 766305 T1819 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 766306 T1819 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 766306 T1819 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 766307 T1819 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 766307 T1819 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 766308 T1819 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 766310 T1819 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 766311 T1819 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 766312 T1819 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 766315 T1819 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 766315 T1819 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 766316 T1819 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=45, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=7.2314453125, floorSegmentMB=1.0478515625, forceMergeDeletesPctAllowed=0.016572250975657177, segmentsPerTier=27.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 766316 T1819 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@a9a903a4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@996e68ab),segFN=segments_1,generation=1}
   [junit4]   2> 766317 T1819 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 766317 T1819 oass.SolrIndexSearcher.<init> Opening Searcher@866076aa[collection1] main
   [junit4]   2> 766317 T1819 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 766319 T1819 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 766319 T1819 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 766320 T1819 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 766320 T1819 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 766320 T1819 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 766321 T1819 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 766322 T1819 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 766323 T1819 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 766325 T1819 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 766326 T1819 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 766326 T1819 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 766328 T1821 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@866076aa[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 766330 T1819 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 766332 T1824 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:48367/_nq/dk collection:control_collection shard:shard1
   [junit4]   2> 766332 T1793 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 766333 T1793 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 766333 T1824 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 766334 T1793 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 766336 T1793 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 766338 T1826 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c0c28182 name:ZooKeeperConnection Watcher:127.0.0.1:35479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 766339 T1793 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 766340 T1793 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 766341 T1793 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 766342 T1824 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 766344 T1820 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 766344 T1824 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 766345 T1793 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 766345 T1824 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C856 name=collection1 org.apache.solr.core.SolrCore@df4fb08f url=http://127.0.0.1:48367/_nq/dk/collection1 node=127.0.0.1:48367__nq%2Fdk C856_STATE=coll:control_collection core:collection1 props:{node_name=127.0.0.1:48367__nq%2Fdk, state=down, core=collection1, base_url=http://127.0.0.1:48367/_nq/dk}
   [junit4]   2> 766346 T1824 C856 P48367 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:48367/_nq/dk/collection1/
   [junit4]   2> 766346 T1824 C856 P48367 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 766347 T1824 C856 P48367 oasc.SyncStrategy.syncToMe http://127.0.0.1:48367/_nq/dk/collection1/ has no replicas
   [junit4]   2> 766347 T1824 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:48367/_nq/dk/collection1/ shard1
   [junit4]   2> 766348 T1824 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 766354 T1820 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 766359 T1820 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 766360 T1827 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 766405 T1824 oasc.ZkController.register We are http://127.0.0.1:48367/_nq/dk/collection1/ and leader is http://127.0.0.1:48367/_nq/dk/collection1/
   [junit4]   2> 766405 T1824 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:48367/_nq/dk
   [junit4]   2> 766406 T1824 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 766406 T1824 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 766406 T1824 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 766408 T1820 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 766412 T1816 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "node_name":"127.0.0.1:48367__nq%2Fdk",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48367/_nq/dk"}
   [junit4]   2> 766514 T1827 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 766514 T1820 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 766622 T1793 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 766623 T1793 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 766625 T1793 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:?????
   [junit4]   2> 766626 T1793 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 766626 T1793 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 766627 T1793 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-003
   [junit4]   2> 766627 T1793 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-003/'
   [junit4]   2> 766635 T1793 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-003/solr.xml
   [junit4]   2> 766668 T1793 oasc.CoreContainer.<init> New CoreContainer 279115171
   [junit4]   2> 766669 T1793 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-003/]
   [junit4]   2> 766670 T1793 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 766671 T1793 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 766671 T1793 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 766672 T1793 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 766672 T1793 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 766672 T1793 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 766673 T1793 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 766673 T1793 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 766674 T1793 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 766675 T1793 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 766675 T1793 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 766676 T1793 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 766676 T1793 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 766677 T1793 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35479/solr
   [junit4]   2> 766677 T1793 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 766678 T1793 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 766679 T1793 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 766680 T1838 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@64eff8d0 name:ZooKeeperConnection Watcher:127.0.0.1:35479 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 766680 T1793 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 766681 T1793 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 766683 T1793 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 766684 T1840 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6c02feb1 name:ZooKeeperConnection Watcher:127.0.0.1:35479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 766684 T1793 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 766692 T1793 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 767697 T1793 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60574__nq%2Fdk
   [junit4]   2> 767699 T1793 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60574__nq%2Fdk
   [junit4]   2> 767702 T1793 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 767710 T1842 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 767710 T1842 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 767711 T1820 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 767711 T1842 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 767715 T1816 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "node_name":"127.0.0.1:60574__nq%2Fdk",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:60574/_nq/dk"}
   [junit4]   2> 767716 T1816 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
   [junit4]   2> 767716 T1816 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 1
   [junit4]   2> 767717 T1816 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 767821 T1841 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 767821 T1827 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 767822 T1820 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 768712 T1842 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 768713 T1842 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 768713 T1842 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 768714 T1842 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 768715 T1842 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 768715 T1842 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-003/collection1/'
   [junit4]   2> 768715 T1842 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 768716 T1842 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 768716 T1842 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 768746 T1842 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 768783 T1842 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 768784 T1842 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 768789 T1842 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 769094 T1842 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 769099 T1842 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 769101 T1842 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 769110 T1842 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 769113 T1842 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 769117 T1842 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 769117 T1842 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 769118 T1842 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 769118 T1842 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 769119 T1842 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 769119 T1842 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 769119 T1842 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 769120 T1842 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 769120 T1842 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-003/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-001/jetty1/
   [junit4]   2> 769120 T1842 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@80ed7c2e
   [junit4]   2> 769121 T1842 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-001/jetty1
   [junit4]   2> 769121 T1842 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-001/jetty1/index/
   [junit4]   2> 769121 T1842 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 769122 T1842 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-001/jetty1/index
   [junit4]   2> 769122 T1842 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=27, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5827239501573246]
   [junit4]   2> 769123 T1842 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@a357dfa1 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@580626d7),segFN=segments_1,generation=1}
   [junit4]   2> 769123 T1842 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 769125 T1842 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 769126 T1842 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 769126 T1842 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 769126 T1842 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 769127 T1842 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 769127 T1842 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 769127 T1842 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 769128 T1842 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 769128 T1842 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 769128 T1842 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 769129 T1842 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 769129 T1842 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 769129 T1842 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 769130 T1842 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 769130 T1842 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 769130 T1842 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 769131 T1842 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 769131 T1842 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 769131 T1842 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 769132 T1842 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 769133 T1842 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 769134 T1842 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 769135 T1842 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 769136 T1842 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 769138 T1842 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 769139 T1842 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 769139 T1842 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=45, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=7.2314453125, floorSegmentMB=1.0478515625, forceMergeDeletesPctAllowed=0.016572250975657177, segmentsPerTier=27.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 769140 T1842 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@a357dfa1 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@580626d7),segFN=segments_1,generation=1}
   [junit4]   2> 769140 T1842 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 769140 T1842 oass.SolrIndexSearcher.<init> Opening Searcher@f8ec7496[collection1] main
   [junit4]   2> 769140 T1842 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 769141 T1842 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 769142 T1842 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 769142 T1842 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 769142 T1842 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 769143 T1842 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 769143 T1842 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 769144 T1842 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 769144 T1842 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 769144 T1842 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 769147 T1843 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@f8ec7496[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 769148 T1842 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 769149 T1846 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:60574/_nq/dk collection:collection1 shard:shard1
   [junit4]   2> 769149 T1793 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 769150 T1793 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 769150 T1846 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 769154 T1846 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 769155 T1820 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 769156 T1846 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 769156 T1846 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C857 name=collection1 org.apache.solr.core.SolrCore@52de84f1 url=http://127.0.0.1:60574/_nq/dk/collection1 node=127.0.0.1:60574__nq%2Fdk C857_STATE=coll:collection1 core:collection1 props:{node_name=127.0.0.1:60574__nq%2Fdk, state=down, core=collection1, base_url=http://127.0.0.1:60574/_nq/dk}
   [junit4]   2> 769157 T1846 C857 P60574 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:60574/_nq/dk/collection1/
   [junit4]   2> 769157 T1846 C857 P60574 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 769157 T1846 C857 P60574 oasc.SyncStrategy.syncToMe http://127.0.0.1:60574/_nq/dk/collection1/ has no replicas
   [junit4]   2> 769158 T1846 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:60574/_nq/dk/collection1/ shard1
   [junit4]   2> 769158 T1846 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 769162 T1820 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 769266 T1827 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 769266 T1841 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 769267 T1820 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 769313 T1846 oasc.ZkController.register We are http://127.0.0.1:60574/_nq/dk/collection1/ and leader is http://127.0.0.1:60574/_nq/dk/collection1/
   [junit4]   2> 769314 T1846 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:60574/_nq/dk
   [junit4]   2> 769314 T1846 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 769315 T1846 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 769315 T1846 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 769317 T1820 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 769320 T1816 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "node_name":"127.0.0.1:60574__nq%2Fdk",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:60574/_nq/dk"}
   [junit4]   2> 769409 T1793 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 769410 T1793 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 769412 T1793 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:?????
   [junit4]   2> 769414 T1793 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 769414 T1793 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 769415 T1793 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-004
   [junit4]   2> 769416 T1793 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-004/'
   [junit4]   2> 769426 T1820 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 769426 T1827 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 769426 T1841 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 769427 T1793 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-004/solr.xml
   [junit4]   2> 769473 T1793 oasc.CoreContainer.<init> New CoreContainer -1965371579
   [junit4]   2> 769473 T1793 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-004/]
   [junit4]   2> 769474 T1793 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 769475 T1793 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 769475 T1793 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 769476 T1793 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 769476 T1793 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 769477 T1793 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 769477 T1793 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 769478 T1793 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 769478 T1793 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 769479 T1793 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 769479 T1793 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 769480 T1793 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 769480 T1793 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 769481 T1793 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35479/solr
   [junit4]   2> 769481 T1793 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 769482 T1793 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 769483 T1793 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 769484 T1857 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9cd2c18a name:ZooKeeperConnection Watcher:127.0.0.1:35479 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 769484 T1793 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 769485 T1793 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 769487 T1793 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 769488 T1859 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ec3fd0df name:ZooKeeperConnection Watcher:127.0.0.1:35479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 769488 T1793 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 769493 T1793 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 770496 T1793 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42321__nq%2Fdk
   [junit4]   2> 770498 T1793 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42321__nq%2Fdk
   [junit4]   2> 770503 T1793 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 770511 T1861 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 770512 T1861 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 770513 T1820 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 770513 T1861 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 770517 T1816 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "node_name":"127.0.0.1:42321__nq%2Fdk",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:42321/_nq/dk"}
   [junit4]   2> 770518 T1816 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
   [junit4]   2> 770518 T1816 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 770622 T1827 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 770622 T1860 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 770622 T1841 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 770622 T1820 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 771515 T1861 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 771515 T1861 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 771516 T1861 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 771516 T1861 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 771518 T1861 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 771519 T1861 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-004/collection1/'
   [junit4]   2> 771520 T1861 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 771521 T1861 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 771521 T1861 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 771554 T1861 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 771590 T1861 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 771600 T1861 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 771606 T1861 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 771881 T1861 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 771885 T1861 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 771888 T1861 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 771899 T1861 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 771902 T1861 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 771906 T1861 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 771906 T1861 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 771907 T1861 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 771907 T1861 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 771908 T1861 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 771908 T1861 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 771908 T1861 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 771909 T1861 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 771909 T1861 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-004/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-001/jetty2/
   [junit4]   2> 771909 T1861 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@80ed7c2e
   [junit4]   2> 771910 T1861 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-001/jetty2
   [junit4]   2> 771910 T1861 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-001/jetty2/index/
   [junit4]   2> 771911 T1861 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 771911 T1861 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-001/jetty2/index
   [junit4]   2> 771911 T1861 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=27, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5827239501573246]
   [junit4]   2> 771912 T1861 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@a74c8619 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@76d3d348),segFN=segments_1,generation=1}
   [junit4]   2> 771912 T1861 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 771915 T1861 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 771915 T1861 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 771916 T1861 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 771916 T1861 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 771916 T1861 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 771916 T1861 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 771917 T1861 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 771917 T1861 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 771917 T1861 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 771918 T1861 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 771918 T1861 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 771919 T1861 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 771919 T1861 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 771919 T1861 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 771920 T1861 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 771920 T1861 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 771920 T1861 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 771921 T1861 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 771921 T1861 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 771921 T1861 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 771923 T1861 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 771924 T1861 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 771925 T1861 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 771926 T1861 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 771928 T1861 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 771929 T1861 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 771930 T1861 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=45, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=7.2314453125, floorSegmentMB=1.0478515625, forceMergeDeletesPctAllowed=0.016572250975657177, segmentsPerTier=27.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 771930 T1861 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@a74c8619 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@76d3d348),segFN=segments_1,generation=1}
   [junit4]   2> 771930 T1861 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 771931 T1861 oass.SolrIndexSearcher.<init> Opening Searcher@34e648ac[collection1] main
   [junit4]   2> 771931 T1861 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 771932 T1861 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 771932 T1861 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 771933 T1861 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 771933 T1861 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 771934 T1861 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 771935 T1861 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 771936 T1861 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 771936 T1861 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 771937 T1861 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 771941 T1862 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@34e648ac[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 771944 T1861 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 771945 T1865 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42321/_nq/dk collection:collection1 shard:shard1
   [junit4]   2> 771946 T1793 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 771947 T1793 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 771950 T1865 oasc.ZkController.register We are http://127.0.0.1:42321/_nq/dk/collection1/ and leader is http://127.0.0.1:60574/_nq/dk/collection1/
   [junit4]   2> 771951 T1865 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42321/_nq/dk
   [junit4]   2> 771951 T1865 oasc.ZkController.checkRecovery Core needs to recover:collection1
   [junit4]   2> 771951 T1865 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
   [junit4]   2> ASYNC  NEW_CORE C858 name=collection1 org.apache.solr.core.SolrCore@96371169 url=http://127.0.0.1:42321/_nq/dk/collection1 node=127.0.0.1:42321__nq%2Fdk C858_STATE=coll:collection1 core:collection1 props:{node_name=127.0.0.1:42321__nq%2Fdk, state=down, core=collection1, base_url=http://127.0.0.1:42321/_nq/dk}
   [junit4]   2> 771952 T1866 C858 P42321 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
   [junit4]   2> 771953 T1866 C858 P42321 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
   [junit4]   2> 771953 T1866 C858 P42321 oasc.RecoveryStrategy.doRecovery Publishing state of core collection1 as recovering, leader is http://127.0.0.1:60574/_nq/dk/collection1/ and I am http://127.0.0.1:42321/_nq/dk/collection1/
   [junit4]   2> 771953 T1866 C858 P42321 oasc.ZkController.publish publishing core=collection1 state=recovering collection=collection1
   [junit4]   2> 771954 T1866 C858 P42321 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 771955 T1820 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 771956 T1866 C858 P42321 oasc.RecoveryStrategy.sendPrepRecoveryCmd Sending prep recovery command to http://127.0.0.1:60574/_nq/dk; WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1%3A42321__nq%252Fdk&coreNodeName=core_node2&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true
   [junit4]   2> 771958 T1834 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
   [junit4]   2> 771958 T1816 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "node_name":"127.0.0.1:42321__nq%2Fdk",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "core_node_name":"core_node2",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:42321/_nq/dk"}
   [junit4]   2> 771962 T1834 oasha.CoreAdminHandler.handleWaitForStateAction Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
   [junit4]   2> 771963 T1834 oasha.CoreAdminHandler.handleWaitForStateAction In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:42321__nq%2Fdk, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"node_name":"127.0.0.1:42321__nq%2Fdk","state":"down","core":"collection1","base_url":"http://127.0.0.1:42321/_nq/dk"}
   [junit4]   2> 772063 T1841 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 772063 T1820 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 772063 T1827 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 772063 T1860 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 772215 T1793 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 772216 T1793 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 772218 T1793 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:?????
   [junit4]   2> 772219 T1793 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 772219 T1793 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 772220 T1793 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-005
   [junit4]   2> 772220 T1793 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-005/'
   [junit4]   2> 772228 T1793 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-005/solr.xml
   [junit4]   2> 772261 T1793 oasc.CoreContainer.<init> New CoreContainer -1629536487
   [junit4]   2> 772262 T1793 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-005/]
   [junit4]   2> 772263 T1793 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 772263 T1793 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 772264 T1793 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 772264 T1793 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 772265 T1793 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 772265 T1793 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 772266 T1793 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 772266 T1793 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 772266 T1793 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 772267 T1793 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 772268 T1793 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 772268 T1793 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 772269 T1793 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 772269 T1793 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35479/solr
   [junit4]   2> 772270 T1793 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 772270 T1793 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 772271 T1793 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 772273 T1878 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a61ba4e name:ZooKeeperConnection Watcher:127.0.0.1:35479 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 772274 T1793 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 772274 T1793 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 772277 T1793 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 772278 T1880 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e11fb246 name:ZooKeeperConnection Watcher:127.0.0.1:35479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 772278 T1793 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 772286 T1793 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 772964 T1834 oasha.CoreAdminHandler.handleWaitForStateAction In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:42321__nq%2Fdk, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"node_name":"127.0.0.1:42321__nq%2Fdk","state":"recovering","core":"collection1","base_url":"http://127.0.0.1:42321/_nq/dk"}
   [junit4]   2> 772965 T1834 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
   [junit4]   2> 772966 T1834 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={state=recovering&onlyIfLeader=true&checkLive=true&onlyIfLeaderActive=true&version=2&core=collection1&wt=javabin&action=PREPRECOVERY&nodeName=127.0.0.1:42321__nq%252Fdk&coreNodeName=core_node2} status=0 QTime=1008 
   [junit4]   2> 773292 T1793 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43197__nq%2Fdk
   [junit4]   2> 773293 T1793 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43197__nq%2Fdk
   [junit4]   2> 773299 T1793 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 773308 T1882 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 773309 T1882 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 773310 T1820 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 773310 T1882 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 773312 T1816 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "node_name":"127.0.0.1:43197__nq%2Fdk",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "operation":"st

[...truncated too long message...]

ascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 816550 T1793 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=3,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=1,adds=1,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=5,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0,transaction_logs_total_size=487,transaction_logs_total_number=1}
   [junit4]   2> 816550 T1793 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 816551 T1793 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 816551 T1793 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2>  C862_STATE=coll:collection1 core:collection1 props:{node_name=127.0.0.1:60606__nq%2Fdk, state=down, core=collection1, base_url=http://127.0.0.1:60606/_nq/dk, leader=true}
   [junit4]   2> 816551 T1793 C862 P60606 oasu.DirectUpdateHandler2.closeWriter Committing on IndexWriter close.
   [junit4]   2> 816555 T1793 C862 P60606 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@e3b1ad20 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@87383f7),segFN=segments_2,generation=2}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@e3b1ad20 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@87383f7),segFN=segments_3,generation=3}
   [junit4]   2> 816556 T1793 C862 P60606 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
   [junit4]   2> 816557 T1793 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 816557 T1793 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 816558 T1793 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-001/jetty4/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-001/jetty4/index;done=false>>]
   [junit4]   2> 816558 T1793 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-001/jetty4/index
   [junit4]   2> 816559 T1793 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-001/jetty4 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-001/jetty4;done=false>>]
   [junit4]   2> 816560 T1793 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001/tempDir-001/jetty4
   [junit4]   2> 816560 T1793 oasc.Overseer.close Overseer (id=92445115034435596-127.0.0.1:60606__nq%2Fdk-n_0000000004) closing
   [junit4]   2> 816561 T1942 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:60606__nq%2Fdk
   [junit4]   2> 818062 T1945 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:347)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:344)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:344)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:346)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:296)
   [junit4]   2> 
   [junit4]   2> 818062 T1941 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 818084 T1793 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/_nq/dk,null}
   [junit4]   2> 823074 T1793 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 823075 T1793 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:35479 35479
   [junit4]   2> 823670 T1794 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:35479 35479
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SyncSliceTest -Dtests.method=testDistribSearch -Dtests.seed=C9793CC7A981939A -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=gu -Dtests.timezone=Europe/Busingen -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 59.4s J0 | SyncSliceTest.testDistribSearch <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: expected:<5> but was:<4>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([C9793CC7A981939A:489FB2DFDEDEF3A6]:0)
   [junit4]    > 	at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:176)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:853)
   [junit4]   2> 823687 T1793 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-C9793CC7A981939A-001
   [junit4]   2> NOTE: test params are: codec=Lucene410: {other_tl1=PostingsFormat(name=Memory doPackFST= true), timestamp=PostingsFormat(name=Memory doPackFST= true), a_i1=PostingsFormat(name=Memory doPackFST= true), id=Lucene41(blocksize=128), text=PostingsFormat(name=SimpleText), rnd_b=Lucene41(blocksize=128), intDefault=PostingsFormat(name=Memory doPackFST= true), range_facet_l=PostingsFormat(name=Memory doPackFST= true), a_t=PostingsFormat(name=Memory doPackFST= true), _version_=PostingsFormat(name=Memory doPackFST= true), multiDefault=PostingsFormat(name=MockRandom)}, docValues:{timestamp=DocValuesFormat(name=Lucene410)}, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=gu, timezone=Europe/Busingen
   [junit4]   2> NOTE: Linux 3.13.0-35-generic x86/IBM Corporation 1.7.0 (32-bit)/cpus=8,threads=1,free=6925544,total=199098368
   [junit4]   2> NOTE: All tests run in this JVM: [HdfsCollectionsAPIDistributedZkTest, TestManagedResource, TestCoreDiscovery, HdfsWriteToMultipleCollectionsTest, ExternalFileFieldSortTest, TestExceedMaxTermLength, SolrCmdDistributorTest, SpellCheckCollatorTest, SSLMigrationTest, IndexSchemaTest, HdfsBasicDistributedZk2Test, FileUtilsTest, TestSolrDeletionPolicy2, BasicDistributedZkTest, ChaosMonkeyNothingIsSafeTest, PluginInfoTest, ParsingFieldUpdateProcessorsTest, NoCacheHeaderTest, TestCustomSort, CursorPagingTest, ChangedSchemaMergeTest, TestSuggestSpellingConverter, TestManagedSchemaDynamicFieldResource, FastVectorHighlighterTest, TestCursorMarkWithoutUniqueKey, CollectionsAPIAsyncDistributedZkTest, DocValuesMissingTest, DocValuesMultiTest, TestRandomDVFaceting, TestFastOutputStream, DistributedSpellCheckComponentTest, HdfsRecoveryZkTest, TestManagedSchema, TestCollectionAPI, TestSchemaSimilarityResource, TestFuzzyAnalyzedSuggestions, DistributedIntervalFacetingTest, TestSolr4Spatial, TestPartialUpdateDeduplication, TestFaceting, TestReRankQParserPlugin, SuggesterTSTTest, ShowFileRequestHandlerTest, TestLeaderElectionZkExpiry, TestQuerySenderListener, SyncSliceTest]
   [junit4] Completed on J0 in 59.45s, 1 test, 1 failure <<< FAILURES!

[...truncated 1084 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:484: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:60: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1358: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:961: There were test failures: 438 suites, 1830 tests, 1 failure, 53 ignored (23 assumptions)

Total time: 109 minutes 41 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_40-ea-b04) - Build # 11235 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11235/
Java: 32bit/jdk1.8.0_40-ea-b04 -client -XX:+UseG1GC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.testDistribSearch

Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:55130/t_/h, http://127.0.0.1:49337/t_/h, http://127.0.0.1:55822/t_/h]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:55130/t_/h, http://127.0.0.1:49337/t_/h, http://127.0.0.1:55822/t_/h]
	at __randomizedtesting.SeedInfo.seed([10F42259FF2B0D2E:9112AC4188746D12]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:319)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:890)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:672)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:615)
	at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.removeAndWaitForLastReplicaGone(DeleteLastCustomShardedReplicaTest.java:116)
	at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.doTest(DeleteLastCustomShardedReplicaTest.java:106)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not  remove replica : customcollreplicadeletion/a/core_node1
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:550)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:340)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:301)
	... 46 more




Build Log:
[...truncated 11930 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/init-core-data-001
   [junit4]   2> 1911237 T5348 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
   [junit4]   2> 1911237 T5348 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /t_/h
   [junit4]   2> 1911244 T5348 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 1911244 T5348 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1911245 T5349 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1911345 T5348 oasc.ZkTestServer.run start zk server on port:39066
   [junit4]   2> 1911346 T5348 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1911346 T5348 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1911349 T5355 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@360167 name:ZooKeeperConnection Watcher:127.0.0.1:39066 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1911349 T5348 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1911350 T5348 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1911350 T5348 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1911353 T5348 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1911354 T5348 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1911355 T5357 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fdf7bc name:ZooKeeperConnection Watcher:127.0.0.1:39066/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1911355 T5348 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1911355 T5348 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1911356 T5348 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1911358 T5348 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1911360 T5348 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1911361 T5348 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1911363 T5348 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1911364 T5348 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1911366 T5348 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1911367 T5348 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1911369 T5348 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1911370 T5348 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1911371 T5348 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1911372 T5348 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1911374 T5348 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1911374 T5348 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1911376 T5348 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1911377 T5348 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1911378 T5348 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1911379 T5348 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1911381 T5348 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1911381 T5348 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1911383 T5348 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1911384 T5348 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1911385 T5348 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1911386 T5348 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1911388 T5348 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1911388 T5348 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1911707 T5348 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1911754 T5348 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55130
   [junit4]   2> 1911755 T5348 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1911756 T5348 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1911757 T5348 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-002
   [junit4]   2> 1911757 T5348 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-002/'
   [junit4]   2> 1911787 T5348 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-002/solr.xml
   [junit4]   2> 1911812 T5348 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-002
   [junit4]   2> 1911812 T5348 oasc.CoreContainer.<init> New CoreContainer 17171561
   [junit4]   2> 1911813 T5348 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-002/]
   [junit4]   2> 1911814 T5348 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1911815 T5348 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1911816 T5348 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1911816 T5348 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1911817 T5348 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1911817 T5348 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1911818 T5348 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1911818 T5348 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1911819 T5348 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1911820 T5348 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1911820 T5348 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1911821 T5348 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1911821 T5348 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1911822 T5348 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39066/solr
   [junit4]   2> 1911823 T5348 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1911823 T5348 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1911824 T5348 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1911825 T5368 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a2fec2 name:ZooKeeperConnection Watcher:127.0.0.1:39066 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1911826 T5348 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1911826 T5348 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1911828 T5348 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1911829 T5370 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e32a8d name:ZooKeeperConnection Watcher:127.0.0.1:39066/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1911829 T5348 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1911831 T5348 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1911833 T5348 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1911836 T5348 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1911838 T5348 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1911841 T5348 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1911844 T5348 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1911845 T5348 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55130_t_%2Fh
   [junit4]   2> 1911846 T5348 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55130_t_%2Fh
   [junit4]   2> 1911847 T5348 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1911849 T5348 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1911850 T5348 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1911852 T5348 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:55130_t_%2Fh
   [junit4]   2> 1911852 T5348 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1911854 T5348 oasc.Overseer.start Overseer (id=92444291248160771-127.0.0.1:55130_t_%2Fh-n_0000000000) starting
   [junit4]   2> 1911856 T5348 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1911862 T5348 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1911863 T5372 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1911863 T5348 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1911865 T5348 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1911867 T5348 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1911869 T5371 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1911871 T5348 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-002
   [junit4]   2> 1911880 T5348 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-002/conf/
   [junit4]   2> 1912078 T5348 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1912079 T5348 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 1912080 T5348 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1912080 T5348 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1912081 T5348 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1912082 T5376 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13bd673 name:ZooKeeperConnection Watcher:127.0.0.1:39066/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1912082 T5348 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1912083 T5348 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1912084 T5348 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1912086 T5348 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1912086 T5348 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 1912087 T5348 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1912087 T5348 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1912088 T5378 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11a8f1 name:ZooKeeperConnection Watcher:127.0.0.1:39066/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1912089 T5348 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1912089 T5348 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1912091 T5379 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1912094 T5371 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: collection1
   [junit4]   2> 1912094 T5371 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 1912094 T5371 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 2
   [junit4]   2> 1912096 T5371 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/collection1/state.json
   [junit4]   2> 1912098 T5379 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1912099 T5380 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1912403 T5348 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 1912403 T5348 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1912406 T5348 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49337
   [junit4]   2> 1912407 T5348 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1912407 T5348 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1912408 T5348 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-003
   [junit4]   2> 1912408 T5348 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-003/'
   [junit4]   2> 1912429 T5348 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-003/solr.xml
   [junit4]   2> 1912448 T5348 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-003
   [junit4]   2> 1912449 T5348 oasc.CoreContainer.<init> New CoreContainer 28927382
   [junit4]   2> 1912449 T5348 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-003/]
   [junit4]   2> 1912450 T5348 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1912450 T5348 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1912451 T5348 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1912451 T5348 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1912451 T5348 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1912452 T5348 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1912452 T5348 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1912452 T5348 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1912453 T5348 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1912453 T5348 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1912454 T5348 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1912454 T5348 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1912455 T5348 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1912455 T5348 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39066/solr
   [junit4]   2> 1912455 T5348 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1912456 T5348 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1912457 T5348 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1912458 T5391 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@141bf4e name:ZooKeeperConnection Watcher:127.0.0.1:39066 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1912458 T5348 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1912459 T5348 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1912460 T5350 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1486d97d55c0006, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1912460 T5348 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1912462 T5393 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1998305 name:ZooKeeperConnection Watcher:127.0.0.1:39066/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1912462 T5348 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1912467 T5348 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1913471 T5348 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49337_t_%2Fh
   [junit4]   2> 1913472 T5348 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49337_t_%2Fh
   [junit4]   2> 1913475 T5348 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1913479 T5348 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-003
   [junit4]   2> 1913487 T5348 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-003/conf/
   [junit4]   2> 1913657 T5348 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1913659 T5348 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 1913659 T5348 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1913937 T5348 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 1913938 T5348 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1913941 T5348 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55822
   [junit4]   2> 1913941 T5348 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1913942 T5348 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1913942 T5348 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-004
   [junit4]   2> 1913943 T5348 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-004/'
   [junit4]   2> 1913962 T5348 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-004/solr.xml
   [junit4]   2> 1913976 T5348 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-004
   [junit4]   2> 1913976 T5348 oasc.CoreContainer.<init> New CoreContainer 12626064
   [junit4]   2> 1913977 T5348 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-004/]
   [junit4]   2> 1913978 T5348 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1913978 T5348 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1913978 T5348 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1913979 T5348 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1913979 T5348 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1913979 T5348 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1913980 T5348 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1913980 T5348 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1913980 T5348 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1913981 T5348 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1913981 T5348 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1913982 T5348 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1913982 T5348 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1913983 T5348 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39066/solr
   [junit4]   2> 1913983 T5348 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1913983 T5348 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1913984 T5348 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1913985 T5406 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9d3401 name:ZooKeeperConnection Watcher:127.0.0.1:39066 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1913985 T5348 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1913986 T5348 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1913987 T5348 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1913988 T5408 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e1a83b name:ZooKeeperConnection Watcher:127.0.0.1:39066/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1913989 T5348 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1913993 T5348 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1914997 T5348 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55822_t_%2Fh
   [junit4]   2> 1914999 T5348 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55822_t_%2Fh
   [junit4]   2> 1915006 T5348 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1915011 T5348 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-004
   [junit4]   2> 1915024 T5348 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-004/conf/
   [junit4]   2> 1915156 T5348 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1915157 T5348 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 1915158 T5348 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1915161 T5348 oasc.AbstractFullDistribZkTestBase.createCollection Creating collection with stateFormat=2: customcollreplicadeletion
   [junit4]   2> 1915162 T5348 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1915162 T5348 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1915164 T5412 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6922a7 name:ZooKeeperConnection Watcher:127.0.0.1:39066/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1915164 T5348 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1915164 T5348 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1915165 T5348 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1915171 T5364 oasha.CollectionsHandler.handleCreateAction Creating Collection : replicationFactor=1&shards=a,b&maxShardsPerNode=5&router.name=implicit&name=customcollreplicadeletion&action=CREATE&numShards=1&stateFormat=2&wt=javabin&version=2
   [junit4]   2> 1915172 T5379 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1915173 T5372 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"customcollreplicadeletion",
   [junit4]   2> 	  "replicationFactor":"1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "maxShardsPerNode":"5",
   [junit4]   2> 	  "shards":"a,b",
   [junit4]   2> 	  "stateFormat":"2",
   [junit4]   2> 	  "router.name":"implicit"}
   [junit4]   2> 1915174 T5413 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"customcollreplicadeletion",
   [junit4]   2> 	  "replicationFactor":"1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "maxShardsPerNode":"5",
   [junit4]   2> 	  "shards":"a,b",
   [junit4]   2> 	  "stateFormat":"2",
   [junit4]   2> 	  "router.name":"implicit"}
   [junit4]   2> 1915176 T5413 oasc.OverseerCollectionProcessor.createConfNode Only one config set found in zk - using it:conf1
   [junit4]   2> 1915176 T5413 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/customcollreplicadeletion 
   [junit4]   2> 1915177 T5413 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion
   [junit4]   2> 1915179 T5379 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1915182 T5371 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: customcollreplicadeletion
   [junit4]   2> 1915182 T5371 oasc.Overseer$ClusterStateUpdater.createCollection Create collection customcollreplicadeletion with shards [a, b]
   [junit4]   2> 1915183 T5371 oasc.Overseer$ClusterStateUpdater.createCollection state version customcollreplicadeletion 2
   [junit4]   2> 1915185 T5371 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 1915186 T5380 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1915186 T5379 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1915186 T5409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1915186 T5394 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1915187 T5414 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1915279 T5413 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection customcollreplicadeletion, shardNames [a, b] , replicationFactor : 1
   [junit4]   2> 1915280 T5413 oasc.OverseerCollectionProcessor.createCollection Creating shard customcollreplicadeletion_a_replica1 as part of slice a of collection customcollreplicadeletion on 127.0.0.1:55130_t_%2Fh
   [junit4]   2> 1915281 T5413 oasc.OverseerCollectionProcessor.createCollection Creating shard customcollreplicadeletion_b_replica1 as part of slice b of collection customcollreplicadeletion on 127.0.0.1:55822_t_%2Fh
   [junit4]   2> 1915285 T5366 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_a_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=a&wt=javabin&version=2
   [junit4]   2> 1915286 T5404 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_b_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=b&wt=javabin&version=2
   [junit4]   2> 1915288 T5366 oasc.ZkController.publish publishing core=customcollreplicadeletion_a_replica1 state=down collection=customcollreplicadeletion
   [junit4]   2> 1915289 T5404 oasc.ZkController.publish publishing core=customcollreplicadeletion_b_replica1 state=down collection=customcollreplicadeletion
   [junit4]   2> 1915290 T5379 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1915291 T5366 oasc.ZkController.preRegister Registering watch for external collection customcollreplicadeletion
   [junit4]   2> 1915291 T5366 oascc.ZkStateReader.addZkWatch addZkWatch customcollreplicadeletion
   [junit4]   2> 1915292 T5404 oasc.ZkController.preRegister Registering watch for external collection customcollreplicadeletion
   [junit4]   2> 1915292 T5404 oascc.ZkStateReader.addZkWatch addZkWatch customcollreplicadeletion
   [junit4]   2> 1915292 T5366 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/customcollreplicadeletion/state.json from ZooKeeper... 
   [junit4]   2> 1915293 T5404 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/customcollreplicadeletion/state.json from ZooKeeper... 
   [junit4]   2> 1915294 T5366 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 0 
   [junit4]   2> 1915295 T5366 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1915295 T5404 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 0 
   [junit4]   2> 1915295 T5404 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1915297 T5371 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 0 
   [junit4]   2> 1915298 T5371 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"customcollreplicadeletion_a_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:55130/t_/h",
   [junit4]   2> 	  "node_name":"127.0.0.1:55130_t_%2Fh",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"a",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1915300 T5371 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 1915301 T5379 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 1915301 T5409 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 1915302 T5409 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 1 
   [junit4]   2> 1915303 T5379 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 1 
   [junit4]   2> 1915306 T5371 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 1 
   [junit4]   2> 1915306 T5371 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"customcollreplicadeletion_b_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:55822/t_/h",
   [junit4]   2> 	  "node_name":"127.0.0.1:55822_t_%2Fh",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"b",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1915308 T5379 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1915309 T5371 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 1915309 T5409 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 1915310 T5379 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 1915310 T5409 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 2 
   [junit4]   2> 1915312 T5379 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 2 
   [junit4]   2> 1916295 T5366 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for customcollreplicadeletion_a_replica1
   [junit4]   2> 1916296 T5404 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for customcollreplicadeletion_b_replica1
   [junit4]   2> 1916296 T5366 oasc.ZkController.createCollectionZkNode Check for collection zkNode:customcollreplicadeletion
   [junit4]   2> 1916296 T5404 oasc.ZkController.createCollectionZkNode Check for collection zkNode:customcollreplicadeletion
   [junit4]   2> 1916297 T5366 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1916298 T5404 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1916298 T5404 oascc.ZkStateReader.readConfigName Load collection config from:/collections/customcollreplicadeletion
   [junit4]   2> 1916298 T5366 oascc.ZkStateReader.readConfigName Load collection config from:/collections/customcollreplicadeletion
   [junit4]   2> 1916299 T5404 oascc.ZkStateReader.readConfigName path=/collections/customcollreplicadeletion configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1916299 T5404 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-004/customcollreplicadeletion_b_replica1/'
   [junit4]   2> 1916304 T5366 oascc.ZkStateReader.readConfigName path=/collections/customcollreplicadeletion configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1916305 T5366 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-002/customcollreplicadeletion_a_replica1/'
   [junit4]   2> 1916353 T5404 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1916373 T5366 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1916381 T5404 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1916382 T5404 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1916404 T5366 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1916405 T5366 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1916461 T5404 oass.IndexSchema.readSchema [customcollreplicadeletion_b_replica1] Schema name=test
   [junit4]   2> 1916473 T5366 oass.IndexSchema.readSchema [customcollreplicadeletion_a_replica1] Schema name=test
   [junit4]   2> 1916686 T5366 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1916694 T5404 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1916694 T5366 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1916696 T5366 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1916700 T5404 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1916702 T5404 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1916718 T5366 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1916723 T5366 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1916724 T5404 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1916726 T5366 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1916728 T5366 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1916728 T5404 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1916728 T5366 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1916729 T5366 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1916730 T5366 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1916731 T5366 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1916731 T5366 oasc.CoreContainer.create Creating SolrCore 'customcollreplicadeletion_a_replica1' using configuration from collection customcollreplicadeletion
   [junit4]   2> 1916732 T5366 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1916732 T5366 oasc.SolrCore.<init> [customcollreplicadeletion_a_replica1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-002/customcollreplicadeletion_a_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-002/customcollreplicadeletion_a_replica1/data/
   [junit4]   2> 1916732 T5404 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1916733 T5366 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@13d14f5
   [junit4]   2> 1916734 T5404 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1916734 T5404 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1916735 T5366 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-002/customcollreplicadeletion_a_replica1/data
   [junit4]   2> 1916735 T5404 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1916735 T5366 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-002/customcollreplicadeletion_a_replica1/data/index/
   [junit4]   2> 1916736 T5366 oasc.SolrCore.initIndex WARN [customcollreplicadeletion_a_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-002/customcollreplicadeletion_a_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1916736 T5404 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1916737 T5404 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1916737 T5366 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-002/customcollreplicadeletion_a_replica1/data/index
   [junit4]   2> 1916738 T5404 oasc.CoreContainer.create Creating SolrCore 'customcollreplicadeletion_b_replica1' using configuration from collection customcollreplicadeletion
   [junit4]   2> 1916738 T5366 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@acb291
   [junit4]   2> 1916739 T5404 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1916740 T5404 oasc.SolrCore.<init> [customcollreplicadeletion_b_replica1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-004/customcollreplicadeletion_b_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-004/customcollreplicadeletion_b_replica1/data/
   [junit4]   2> 1916740 T5366 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@3ed321 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@f5c9be),segFN=segments_1,generation=1}
   [junit4]   2> 1916740 T5404 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@13d14f5
   [junit4]   2> 1916741 T5366 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1916742 T5404 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-004/customcollreplicadeletion_b_replica1/data
   [junit4]   2> 1916743 T5404 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-004/customcollreplicadeletion_b_replica1/data/index/
   [junit4]   2> 1916743 T5404 oasc.SolrCore.initIndex WARN [customcollreplicadeletion_b_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-004/customcollreplicadeletion_b_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1916744 T5404 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-004/customcollreplicadeletion_b_replica1/data/index
   [junit4]   2> 1916745 T5404 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@4fe887
   [junit4]   2> 1916746 T5404 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@eb650d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1cd2b0e),segFN=segments_1,generation=1}
   [junit4]   2> 1916747 T5404 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1916752 T5366 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1916753 T5366 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1916754 T5366 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1916754 T5366 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1916755 T5366 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1916755 T5366 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1916756 T5366 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1916757 T5366 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1916757 T5404 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1916758 T5366 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1916758 T5404 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1916759 T5404 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1916759 T5366 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1916760 T5404 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1916761 T5404 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1916761 T5366 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1916761 T5404 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1916763 T5366 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1916763 T5404 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1916764 T5404 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1916764 T5404 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1916764 T5366 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1916765 T5404 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1916765 T5366 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1916766 T5404 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1916766 T5366 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1916767 T5404 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1916767 T5366 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1916768 T5404 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1916768 T5366 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1916768 T5404 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1916769 T5366 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1916769 T5404 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1916770 T5404 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1916770 T5366 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1916771 T5404 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1916771 T5404 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1916772 T5366 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1916772 T5404 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1916773 T5404 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1916775 T5404 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1916775 T5366 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1916778 T5404 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1916778 T5366 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1916780 T5404 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1916781 T5366 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1916781 T5404 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1916783 T5366 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1916784 T5404 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1916784 T5404 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1916785 T5404 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=45, maxMergeAtOnceExplicit=27, maxMergedSegmentMB=26.0888671875, floorSegmentMB=0.5869140625, forceMergeDeletesPctAllowed=7.958769426577641, segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1916786 T5366 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1916786 T5404 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@eb650d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1cd2b0e),segFN=segments_1,generation=1}
   [junit4]   2> 1916787 T5404 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1916787 T5366 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1916788 T5404 oass.SolrIndexSearcher.<init> Opening Searcher@140cd44[customcollreplicadeletion_b_replica1] main
   [junit4]   2> 1916788 T5404 oascc.ZkStateReader.readConfigName Load collection config from:/collections/customcollreplicadeletion
   [junit4]   2> 1916789 T5366 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=45, maxMergeAtOnceExplicit=27, maxMergedSegmentMB=26.0888671875, floorSegmentMB=0.5869140625, forceMergeDeletesPctAllowed=7.958769426577641, segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1916789 T5404 oascc.ZkStateReader.readConfigName path=/collections/customcollreplicadeletion configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1916790 T5404 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1916790 T5366 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@3ed321 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@f5c9be),segFN=segments_1,generation=1}
   [junit4]   2> 1916790 T5404 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1916791 T5404 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1916791 T5366 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1916792 T5404 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1916792 T5366 oass.SolrIndexSearcher.<init> Opening Searcher@268eaa[customcollreplicadeletion_a_replica1] main
   [junit4]   2> 1916793 T5404 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1916793 T5366 oascc.ZkStateReader.readConfigName Load collection config from:/collections/customcollreplicadeletion
   [junit4]   2> 1916793 T5404 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1916794 T5366 oascc.ZkStateReader.readConfigName path=/collections/customcollreplicadeletion configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1916795 T5366 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1916794 T5404 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 1916795 T5366 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1916796 T5366 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1916797 T5366 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1916797 T5404 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1916797 T5404 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1916798 T5366 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1916798 T5404 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1916799 T5366 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1916799 T5366 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 1916800 T5366 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1916803 T5417 oasc.SolrCore.registerSearcher [customcollreplicadeletion_b_replica1] Registered new searcher Searcher@140cd44[customcollreplicadeletion_b_replica1] main{UninvertingDirectoryReader()}
   [junit4]   2> 1916806 T5418 oasc.SolrCore.registerSearcher [customcollreplicadeletion_a_replica1] Registered new searcher Searcher@268eaa[customcollreplicadeletion_a_replica1] main{UninvertingDirectoryReader()}
   [junit4]   2> 1916810 T5404 oasc.CoreContainer.registerCore registering core: customcollreplicadeletion_b_replica1
   [junit4]   2> 1916810 T5404 oasc.ZkController.register Register replica - core:customcollreplicadeletion_b_replica1 address:http://127.0.0.1:55822/t_/h collection:customcollreplicadeletion shard:b
   [junit4]   2> 1916811 T5404 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion/leader_elect/b/election
   [junit4]   2> 1916815 T5366 oasc.CoreContainer.registerCore registering core: customcollreplicadeletion_a_replica1
   [junit4]   2> 1916816 T5404 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard b
   [junit4]   2> 1916816 T5366 oasc.ZkController.register Register replica - core:customcollreplicadeletion_a_replica1 address:http://127.0.0.1:55130/t_/h collection:customcollreplicadeletion shard:a
   [junit4]   2> 1916817 T5379 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1916818 T5404 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1916818 T5404 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 1916817 T5366 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion/leader_elect/a/election
   [junit4]   2> ASYNC  NEW_CORE C1146 name=customcollreplicadeletion_b_replica1 org.apache.solr.core.SolrCore@14347c5 url=http://127.0.0.1:55822/t_/h/customcollreplicadeletion_b_replica1 node=127.0.0.1:55822_t_%2Fh C1146_STATE=coll:customcollreplicadeletion core:customcollreplicadeletion_b_replica1 props:{core=customcollreplicadeletion_b_replica1, base_url=http://127.0.0.1:55822/t_/h, node_name=127.0.0.1:55822_t_%2Fh, state=down}
   [junit4]   2> 1916818 T5404 C1146 P55822 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:55822/t_/h/customcollreplicadeletion_b_replica1/
   [junit4]   2> 1916819 T5404 C1146 P55822 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1916820 T5404 C1146 P55822 oasc.SyncStrategy.syncToMe http://127.0.0.1:55822/t_/h/customcollreplicadeletion_b_replica1/ has no replicas
   [junit4]   2> 1916820 T5404 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:55822/t_/h/customcollreplicadeletion_b_replica1/ b
   [junit4]   2> 1916821 T5404 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion/leaders/b
   [junit4]   2> 1916823 T5371 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 2 
   [junit4]   2> 1916824 T5366 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard a
   [junit4]   2> 1916825 T5371 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 1916826 T5409 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 1916826 T5379 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 1916827 T5409 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 3 
   [junit4]   2> 1916827 T5366 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1916828 T5366 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1147 name=customcollreplicadeletion_a_replica1 org.apache.solr.core.SolrCore@bf1531 url=http://127.0.0.1:55130/t_/h/customcollreplicadeletion_a_replica1 node=127.0.0.1:55130_t_%2Fh C1147_STATE=coll:customcollreplicadeletion core:customcollreplicadeletion_a_replica1 props:{core=customcollreplicadeletion_a_replica1, base_url=http://127.0.0.1:55130/t_/h, node_name=127.0.0.1:55130_t_%2Fh, state=down}
   [junit4]   2> 1916828 T5366 C1147 P55130 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:55130/t_/h/customcollreplicadeletion_a_replica1/
   [junit4]   2> 1916828 T5379 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 3 
   [junit4]   2> 1916829 T5366 C1147 P55130 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1916830 T5366 C1147 P55130 oasc.SyncStrategy.syncToMe http://127.0.0.1:55130/t_/h/customcollreplicadeletion_a_replica1/ has no replicas
   [junit4]   2> 1916830 T5366 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:55130/t_/h/customcollreplicadeletion_a_replica1/ a
   [junit4]   2> 1916831 T5366 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion/leaders/a
   [junit4]   2> 1916832 T5371 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 3 
   [junit4]   2> 1916834 T5379 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1916835 T5371 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 1916835 T5409 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 1916835 T5379 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 1916836 T5409 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 4 
   [junit4]   2> 1916838 T5379 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 4 
   [junit4]   2> 1916841 T5371 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 4 
   [junit4]   2> 1916843 T5379 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1916843 T5371 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 1916844 T5409 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 1916844 T5379 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 1916845 T5409 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 5 
   [junit4]   2> 1916846 T5379 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 5 
   [junit4]   2> 1916849 T5371 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 5 
   [junit4]   2> 1916851 T5379 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1916852 T5371 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 1916852 T5409 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 1916852 T5379 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 1916853 T5409 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 6 
   [junit4]   2> 1916855 T5379 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 6 
   [junit4]   2> 1916875 T5404 oasc.ZkController.register We are http://127.0.0.1:55822/t_/h/customcollreplicadeletion_b_replica1/ and leader is http://127.0.0.1:55822/t_/h/customcollreplicadeletion_b_replica1/
   [junit4]   2> 1916876 T5404 oasc.ZkController.register No LogReplay needed for core=customcollreplicadeletion_b_replica1 baseURL=http://127.0.0.1:55822/t_/h
   [junit4]   2> 1916876 T5404 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1916877 T5404 oasc.ZkController.publish publishing core=customcollreplicadeletion_b_replica1 state=active collection=customcollreplicadeletion
   [junit4]   2> 1916878 T5379 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1916880 T5404 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 6 
   [junit4]   2> 1916881 T5404 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_b_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=b&wt=javabin&version=2} status=0 QTime=1595 
   [junit4]   2> 1916882 T5371 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 6 
   [junit4]   2> 1916882 T5371 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"customcollreplicadeletion_b_replica1",
   [junit4]   2> 	  "core_node_name":"core_node2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:55822/t_/h",
   [junit4]   2> 	  "node_name":"127.0.0.1:55822_t_%2Fh",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"b",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1916884 T5371 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 1916885 T5409 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 1916886 T5379 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 1916886 T5409 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 7 
   [junit4]   2> 1916886 T5366 oasc.ZkController.register We are http://127.0.0.1:55130/t_/h/customcollreplicadeletion_a_replica1/ and leader is http://127.0.0.1:55130/t_/h/customcollreplicadeletion_a_replica1/
   [junit4]   2> 1916887 T5366 oasc.ZkController.register No LogReplay needed for core=customcollreplicadeletion_a_replica1 baseURL=http://127.0.0.1:55130/t_/h
   [junit4]   2> 1916887 T5366 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1916887 T5379 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 7 
   [junit4]   2> 1916888 T5366 oasc.ZkController.publish publishing core=customcollreplicadeletion_a_replica1 state=active collection=customcollreplicadeletion
   [junit4]   2> 1916889 T5379 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1916892 T5366 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 7 
   [junit4]   2> 1916893 T5366 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_a_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=a&wt=javabin&version=2} status=0 QTime=1608 
   [junit4]   2> 1916894 T5413 oasc.OverseerCollectionProcessor.createCollection Finished create command on all shards for collection: customcollreplicadeletion
   [junit4]   2> 1916896 T5413 oasc.OverseerCollectionProcessor$Runner.run Overseer Collection Processor: Message id:/overseer/collection-queue-work/qn-0000000000 complete, response:{success={null={responseHeader={status=0,QTime=1595},core=customcollreplicadeletion_b_replica1},null={responseHeader={status=0,QTime=1608},core=customcollreplicadeletion_a_replica1}}}
   [junit4]   2> 1916896 T5371 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 7 
   [junit4]   2> 1916896 T5371 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"customcollreplicadeletion_a_replica1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:55130/t_/h",
   [junit4]   2> 	  "node_name":"127.0.0.1:55130_t_%2Fh",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"a",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1916896 T5379 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work/qnr-0000000000 state: SyncConnected type NodeDataChanged
   [junit4]   2> 1916897 T5379 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1916899 T5364 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={replicationFactor=1&shards=a,b&maxShardsPerNode=5&router.name=implicit&name=customcollreplicadeletion&action=CREATE&numShards=1&stateFormat=2&wt=javabin&version=2} status=0 QTime=1728 
   [junit4]   2> 1916900 T5348 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: customcollreplicadeletion failOnTimeout:true timeout (sec):330
   [junit4]   2> 1916900 T5371 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 1916901 T5409 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 1916901 T5379 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 1916902 T5409 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 8 
   [junit4]   2> 1916903 T5379 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 8 
   [junit4]   2> 1916904 T5348 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: customcollreplicadeletion
   [junit4]   2> 1916904 T5348 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1916905 T5348 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1916907 T5422 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@168a57 name:ZooKeeperConnection Watcher:127.0.0.1:39066/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1916907 T5348 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1916907 T5348 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1916908 T5348 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1916915 T5366 oasha.CollectionsHandler.handleRemoveReplica Remove replica: replica=core_node1&action=deletereplica&collection=customcollreplicadeletion&shard=a&wt=javabin&version=2
   [junit4]   2> 1916917 T5379 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1916918 T5372 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000002 message:{
   [junit4]   2> 	  "operation":"deletereplica",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "shard":"a",
   [junit4]   2> 	  "replica":"core_node1"}
   [junit4]   2> 1916918 T5423 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : deletereplica , {
   [junit4]   2> 	  "operation":"deletereplica",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "shard":"a",
   [junit4]   2> 	  "replica":"core_node1"}
   [junit4]   2> 1916921 T5363 oasc.SolrCore.close [customcollreplicadeletion_a_replica1]  CLOSING SolrCore org.apache.solr.core.SolrCore@bf1531
   [junit4]   2> 1916941 T5363 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
   [junit4]   2> 1916942 T5363 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 1916942 T5363 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1916943 T5363 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 1916944 T5363 oasc.SolrCore.closeSearcher [customcollreplicadeletion_a_replica1] Closing main searcher on request.
   [junit4]   2> 1916945 T5363 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1916945 T5363 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001/tempDir-002/customcollreplicadeletion_a_replica1/data/index [CachedDir<<refC

[...truncated too long message...]

g.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:347)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:344)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:344)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:346)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:296)
   [junit4]   2> 
   [junit4]   2> 2040299 T5348 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 2040300 T5348 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:39066 39066
   [junit4]   2> 2041011 T5349 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:39066 39066
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DeleteLastCustomShardedReplicaTest -Dtests.method=testDistribSearch -Dtests.seed=10F42259FF2B0D2E -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ja_JP_JP_#u-ca-japanese -Dtests.timezone=Africa/Kigali -Dtests.file.encoding=UTF-8
   [junit4] ERROR    130s J0 | DeleteLastCustomShardedReplicaTest.testDistribSearch <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:55130/t_/h, http://127.0.0.1:49337/t_/h, http://127.0.0.1:55822/t_/h]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([10F42259FF2B0D2E:9112AC4188746D12]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:319)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:890)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:672)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:615)
   [junit4]    > 	at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.removeAndWaitForLastReplicaGone(DeleteLastCustomShardedReplicaTest.java:116)
   [junit4]    > 	at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.doTest(DeleteLastCustomShardedReplicaTest.java:106)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not  remove replica : customcollreplicadeletion/a/core_node1
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:550)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:340)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:301)
   [junit4]    > 	... 46 more
   [junit4]   2> 2041130 T5348 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.DeleteLastCustomShardedReplicaTest-10F42259FF2B0D2E-001
   [junit4]   2> 129900 T5347 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=ja_JP_JP_#u-ca-japanese, timezone=Africa/Kigali
   [junit4]   2> NOTE: Linux 3.13.0-35-generic i386/Oracle Corporation 1.8.0_40-ea (32-bit)/cpus=8,threads=1,free=63827304,total=377487360
   [junit4]   2> NOTE: All tests run in this JVM: [DistribDocExpirationUpdateProcessorTest, DeleteReplicaTest, TriLevelCompositeIdRoutingTest, PeerSyncTest, SignatureUpdateProcessorFactoryTest, TestFaceting, TestSchemaResource, TestTrie, JsonLoaderTest, SolrCoreCheckLockOnStartupTest, TestExceedMaxTermLength, TestAnalyzedSuggestions, TestFastOutputStream, CursorPagingTest, DisMaxRequestHandlerTest, BadComponentTest, TestWordDelimiterFilterFactory, TestFuzzyAnalyzedSuggestions, SyncSliceTest, HdfsDirectoryTest, HdfsRecoveryZkTest, DocValuesMultiTest, TestJmxIntegration, TestAnalyzeInfixSuggestions, OutOfBoxZkACLAndCredentialsProvidersTest, BasicDistributedZkTest, BJQParserTest, TestComponentsName, ExternalFileFieldSortTest, TestShardHandlerFactory, TestManagedSchemaFieldTypeResource, HdfsUnloadDistributedZkTest, DirectUpdateHandlerTest, DocumentAnalysisRequestHandlerTest, AbstractAnalyticsStatsTest, TestDocSet, TestSolrDeletionPolicy1, DirectSolrSpellCheckerTest, TestManagedResource, TestChildDocTransformer, TestSurroundQueryParser, TestManagedSynonymFilterFactory, SuggesterTSTTest, ExpressionTest, TestSchemaSimilarityResource, LeaderElectionTest, SystemInfoHandlerTest, DocumentBuilderTest, RecoveryZkTest, QueryResultKeyTest, NoFacetTest, TestStressReorder, TestSolrCoreProperties, TestRandomDVFaceting, XmlUpdateRequestHandlerTest, AsyncMigrateRouteKeyTest, AbstractAnalyticsFacetTest, ModifyConfFileTest, PathHierarchyTokenizerFactoryTest, TestRemoteStreaming, TestSolrDeletionPolicy2, HighlighterMaxOffsetTest, HdfsBasicDistributedZk2Test, TestDFRSimilarityFactory, SolrIndexSplitterTest, CursorMarkTest, SpellingQueryConverterTest, TestSystemIdResolver, StatelessScriptUpdateProcessorFactoryTest, ResponseLogComponentTest, TestStressUserVersions, TestReloadAndDeleteDocs, TestFreeTextSuggestions, ChangedSchemaMergeTest, TestLMDirichletSimilarityFactory, SpellPossibilityIteratorTest, FieldFacetTest, JSONWriterTest, TestMiniSolrCloudCluster, TestRangeQuery, TestSuggestSpellingConverter, TimeZoneUtilsTest, MinimalSchemaTest, ReturnFieldsTest, SuggesterTest, OverseerCollectionProcessorTest, RollingRestartTest, StandardRequestHandlerTest, StressHdfsTest, TestRTGBase, HighlighterConfigTest, DistributedDebugComponentTest, TestArbitraryIndexDir, TestHashPartitioner, ClusterStateUpdateTest, DeleteShardTest, TestSolrIndexConfig, TestCopyFieldCollectionResource, IndexSchemaTest, DocValuesMissingTest, TestRawResponseWriter, TestLMJelinekMercerSimilarityFactory, TestDynamicFieldResource, TestReplicationHandler, RegexBoostProcessorTest, DistributedSpellCheckComponentTest, TestSort, ShardRoutingTest, ZkCLITest, TestDistributedSearch, AutoCommitTest, TestRecovery, TermVectorComponentDistributedTest, TestReload, TestStressVersions, HardAutoCommitTest, DistributedTermsComponentTest, QueryElevationComponentTest, TestFiltering, TestCoreDiscovery, AnalysisAfterCoreReloadTest, TestExtendedDismaxParser, SpellCheckCollatorTest, CoreAdminHandlerTest, SolrRequestParserTest, SpatialFilterTest, NoCacheHeaderTest, TestAtomicUpdateErrorCases, SolrInfoMBeanTest, XsltUpdateRequestHandlerTest, LukeRequestHandlerTest, TestQueryTypes, TestOmitPositions, TestIndexingPerformance, RequiredFieldsTest, SolrPluginUtilsTest, MBeansHandlerTest, TestPartialUpdateDeduplication, TestPHPSerializedResponseWriter, SearchHandlerTest, TestLFUCache, TestQuerySenderListener, SolrIndexConfigTest, SampleTest, TestBinaryField, TestSearchPerf, TestElisionMultitermQuery, TestConfig, TestPostingsSolrHighlighter, DirectSolrConnectionTest, NotRequiredUniqueKeyTest, TestLuceneMatchVersion, TestCodecSupport, TestPerFieldSimilarity, TestDefaultSimilarityFactory, OpenExchangeRatesOrgProviderTest, TestFastLRUCache, DateFieldTest, DOMUtilTest, ClusterStateTest, TestSolrJ, TestLRUCache, TestUtils, TestDocumentBuilder, ZkNodePropsTest, SliceStateTest, UUIDFieldTest, FileUtilsTest, SolrTestCaseJ4Test, TestCursorMarkWithoutUniqueKey, TestDistributedMissingSort, TestTolerantSearch, QueryFacetTest, AliasIntegrationTest, DeleteLastCustomShardedReplicaTest]
   [junit4] Completed on J0 in 130.20s, 1 test, 1 error <<< FAILURES!

[...truncated 290 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:471: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:60: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1358: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:961: There were test failures: 438 suites, 1830 tests, 1 error, 45 ignored (23 assumptions)

Total time: 103 minutes 21 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_40-ea-b04 -client -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_67) - Build # 11234 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11234/
Java: 32bit/jdk1.7.0_67 -server -XX:+UseSerialGC

All tests passed

Build Log:
[...truncated 31754 lines...]
-check-forbidden-all:
[forbidden-apis] Reading bundled API signatures: jdk-unsafe-1.7
[forbidden-apis] Reading bundled API signatures: jdk-deprecated-1.7
[forbidden-apis] Reading bundled API signatures: commons-io-unsafe-2.3
[forbidden-apis] Reading API signatures: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/tools/forbiddenApis/base.txt
[forbidden-apis] Reading API signatures: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/tools/forbiddenApis/servlet-api.txt
[forbidden-apis] Loading classes to check...
[forbidden-apis] Scanning for API signatures and dependencies...
[forbidden-apis] Forbidden method invocation: java.lang.String#<init>(byte[]) [Uses default charset]
[forbidden-apis]   in org.apache.solr.cloud.Overseer$ClusterStateUpdater (Overseer.java:316)
[forbidden-apis] Scanned 2357 (and 1559 related) class file(s) for forbidden API invocations (in 1.55s), 1 error(s).

BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:85: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:271: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:479: Check for forbidden API calls failed, see log.

Total time: 105 minutes 28 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.7.0_67 -server -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_20) - Build # 11233 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11233/
Java: 64bit/jdk1.8.0_20 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.SyncSliceTest.testDistribSearch

Error Message:
expected:<5> but was:<4>

Stack Trace:
java.lang.AssertionError: expected:<5> but was:<4>
	at __randomizedtesting.SeedInfo.seed([E071A2599265EA36:61972C41E53A8A0A]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:176)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11190 lines...]
   [junit4] Suite: org.apache.solr.cloud.SyncSliceTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/init-core-data-001
   [junit4]   2> 616502 T1773 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
   [junit4]   2> 616502 T1773 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /lm_wky/ic
   [junit4]   2> 616505 T1773 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 616505 T1773 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 616506 T1774 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 616605 T1773 oasc.ZkTestServer.run start zk server on port:40048
   [junit4]   2> 616606 T1773 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 616606 T1773 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 616608 T1780 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@752f9e82 name:ZooKeeperConnection Watcher:127.0.0.1:40048 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 616608 T1773 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 616608 T1773 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 616608 T1773 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 616610 T1773 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 616611 T1773 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 616611 T1782 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@66b742ef name:ZooKeeperConnection Watcher:127.0.0.1:40048/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 616611 T1773 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 616612 T1773 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 616612 T1773 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 616613 T1773 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 616614 T1773 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 616615 T1773 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 616616 T1773 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 616616 T1773 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 616618 T1773 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 616618 T1773 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 616620 T1773 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 616620 T1773 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 616622 T1773 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 616622 T1773 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 616623 T1773 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 616623 T1773 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 616625 T1773 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 616625 T1773 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 616626 T1773 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 616627 T1773 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 616627 T1773 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 616628 T1773 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 616629 T1773 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 616629 T1773 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 616630 T1773 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 616630 T1773 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 616631 T1773 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 616632 T1773 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 616875 T1773 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 616890 T1773 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57674
   [junit4]   2> 616890 T1773 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 616891 T1773 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 616891 T1773 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-002
   [junit4]   2> 616891 T1773 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-002/'
   [junit4]   2> 616914 T1773 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-002/solr.xml
   [junit4]   2> 616921 T1773 oasc.CoreContainer.<init> New CoreContainer 559068881
   [junit4]   2> 616921 T1773 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-002/]
   [junit4]   2> 616922 T1773 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 616922 T1773 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 616922 T1773 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 616923 T1773 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 616923 T1773 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 616923 T1773 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 616923 T1773 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 616923 T1773 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 616924 T1773 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 616924 T1773 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 616924 T1773 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 616925 T1773 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 616925 T1773 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 616925 T1773 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40048/solr
   [junit4]   2> 616925 T1773 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 616926 T1773 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 616926 T1773 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 616928 T1793 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19370acb name:ZooKeeperConnection Watcher:127.0.0.1:40048 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 616928 T1773 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 616928 T1773 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 616930 T1773 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 616931 T1795 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ab2687 name:ZooKeeperConnection Watcher:127.0.0.1:40048/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 616931 T1773 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 616932 T1773 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 616933 T1773 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 616935 T1773 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 616936 T1773 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 616938 T1773 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 616939 T1773 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 616940 T1773 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57674_lm_wky%2Fic
   [junit4]   2> 616940 T1773 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57674_lm_wky%2Fic
   [junit4]   2> 616941 T1773 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 616942 T1773 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 616943 T1773 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 616944 T1773 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:57674_lm_wky%2Fic
   [junit4]   2> 616945 T1773 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 616946 T1773 oasc.Overseer.start Overseer (id=92442518116171779-127.0.0.1:57674_lm_wky%2Fic-n_0000000000) starting
   [junit4]   2> 616947 T1773 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 616951 T1773 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 616952 T1797 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 616952 T1773 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 616954 T1773 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 616954 T1773 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 616956 T1796 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 616957 T1799 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 616957 T1799 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 616958 T1799 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 616958 T1800 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 616959 T1796 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57674/lm_wky/ic",
   [junit4]   2> 	  "node_name":"127.0.0.1:57674_lm_wky%2Fic",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 616959 T1796 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 616959 T1796 oasc.Overseer$ClusterStateUpdater.createCollection state version control_collection 1
   [junit4]   2> 616959 T1796 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 616960 T1800 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 617958 T1799 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 617958 T1799 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 617959 T1799 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 617959 T1799 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 617960 T1799 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 617960 T1799 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-002/collection1/'
   [junit4]   2> 617961 T1799 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 617961 T1799 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 617961 T1799 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 617983 T1799 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 618005 T1799 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 618006 T1799 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 618010 T1799 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 618113 T1799 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 618116 T1799 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 618117 T1799 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 618124 T1799 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 618126 T1799 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 618127 T1799 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 618127 T1799 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 618128 T1799 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 618128 T1799 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 618128 T1799 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 618129 T1799 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 618129 T1799 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 618129 T1799 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 618129 T1799 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-001/control/data/
   [junit4]   2> 618129 T1799 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@706aca0d
   [junit4]   2> 618130 T1799 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-001/control/data
   [junit4]   2> 618130 T1799 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-001/control/data/index/
   [junit4]   2> 618130 T1799 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 618130 T1799 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-001/control/data/index
   [junit4]   2> 618131 T1799 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1318373034, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 618131 T1799 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@6e3c2034 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3937dcf2),segFN=segments_1,generation=1}
   [junit4]   2> 618132 T1799 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 618134 T1799 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 618134 T1799 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 618134 T1799 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 618134 T1799 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 618135 T1799 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 618135 T1799 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 618135 T1799 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 618135 T1799 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 618135 T1799 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 618136 T1799 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 618136 T1799 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 618136 T1799 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 618136 T1799 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 618136 T1799 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 618136 T1799 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 618137 T1799 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 618137 T1799 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 618137 T1799 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 618137 T1799 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 618137 T1799 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 618139 T1799 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 618139 T1799 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 618140 T1799 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 618141 T1799 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 618142 T1799 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 618142 T1799 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 618142 T1799 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=10, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 618143 T1799 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@6e3c2034 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3937dcf2),segFN=segments_1,generation=1}
   [junit4]   2> 618143 T1799 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 618143 T1799 oass.SolrIndexSearcher.<init> Opening Searcher@126677fc[collection1] main
   [junit4]   2> 618143 T1799 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 618144 T1799 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 618144 T1799 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 618144 T1799 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 618144 T1799 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 618144 T1799 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 618145 T1799 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 618145 T1799 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 618145 T1799 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 618147 T1799 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 618147 T1799 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 618147 T1799 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 618148 T1801 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@126677fc[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 618150 T1799 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 618151 T1804 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:57674/lm_wky/ic collection:control_collection shard:shard1
   [junit4]   2> 618151 T1773 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 618151 T1773 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 618151 T1804 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 618152 T1773 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 618152 T1773 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 618154 T1806 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7108647f name:ZooKeeperConnection Watcher:127.0.0.1:40048/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 618154 T1773 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 618154 T1773 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 618155 T1773 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 618155 T1804 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 618157 T1800 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 618157 T1804 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 618157 T1804 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 618158 T1773 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> ASYNC  NEW_CORE C748 name=collection1 org.apache.solr.core.SolrCore@d3ba5ee url=http://127.0.0.1:57674/lm_wky/ic/collection1 node=127.0.0.1:57674_lm_wky%2Fic C748_STATE=coll:control_collection core:collection1 props:{core=collection1, base_url=http://127.0.0.1:57674/lm_wky/ic, node_name=127.0.0.1:57674_lm_wky%2Fic, state=down}
   [junit4]   2> 618158 T1804 C748 P57674 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:57674/lm_wky/ic/collection1/
   [junit4]   2> 618158 T1804 C748 P57674 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 618158 T1804 C748 P57674 oasc.SyncStrategy.syncToMe http://127.0.0.1:57674/lm_wky/ic/collection1/ has no replicas
   [junit4]   2> 618158 T1804 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:57674/lm_wky/ic/collection1/ shard1
   [junit4]   2> 618159 T1804 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 618162 T1800 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 618264 T1800 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 618265 T1807 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 618312 T1804 oasc.ZkController.register We are http://127.0.0.1:57674/lm_wky/ic/collection1/ and leader is http://127.0.0.1:57674/lm_wky/ic/collection1/
   [junit4]   2> 618313 T1804 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:57674/lm_wky/ic
   [junit4]   2> 618313 T1804 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 618313 T1804 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 618314 T1804 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 618315 T1800 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 618318 T1796 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57674/lm_wky/ic",
   [junit4]   2> 	  "node_name":"127.0.0.1:57674_lm_wky%2Fic",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 618420 T1807 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 618420 T1800 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 618433 T1773 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 618434 T1773 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 618437 T1773 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:48650
   [junit4]   2> 618438 T1773 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 618439 T1773 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 618439 T1773 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-003
   [junit4]   2> 618439 T1773 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-003/'
   [junit4]   2> 618451 T1773 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-003/solr.xml
   [junit4]   2> 618467 T1773 oasc.CoreContainer.<init> New CoreContainer 721098857
   [junit4]   2> 618467 T1773 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-003/]
   [junit4]   2> 618467 T1773 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 618468 T1773 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 618468 T1773 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 618468 T1773 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 618468 T1773 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 618469 T1773 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 618469 T1773 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 618469 T1773 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 618469 T1773 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 618470 T1773 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 618470 T1773 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 618470 T1773 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 618471 T1773 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 618471 T1773 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40048/solr
   [junit4]   2> 618471 T1773 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 618471 T1773 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 618472 T1773 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 618473 T1818 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4fa5662c name:ZooKeeperConnection Watcher:127.0.0.1:40048 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 618474 T1773 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 618474 T1773 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 618476 T1773 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 618477 T1820 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6b4838c3 name:ZooKeeperConnection Watcher:127.0.0.1:40048/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 618477 T1773 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 618483 T1773 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 619485 T1773 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48650_lm_wky%2Fic
   [junit4]   2> 619487 T1773 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48650_lm_wky%2Fic
   [junit4]   2> 619491 T1773 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 619496 T1822 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 619496 T1822 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 619497 T1800 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 619497 T1822 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 619500 T1796 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48650/lm_wky/ic",
   [junit4]   2> 	  "node_name":"127.0.0.1:48650_lm_wky%2Fic",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 619500 T1796 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
   [junit4]   2> 619500 T1796 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 1
   [junit4]   2> 619501 T1796 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 619604 T1807 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 619604 T1800 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 619604 T1821 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 620498 T1822 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 620498 T1822 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 620499 T1822 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 620499 T1822 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 620500 T1822 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 620500 T1822 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-003/collection1/'
   [junit4]   2> 620501 T1822 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 620502 T1822 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 620502 T1822 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 620528 T1822 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 620538 T1822 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 620540 T1822 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 620543 T1822 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 620661 T1822 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 620667 T1822 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 620669 T1822 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 620683 T1822 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 620686 T1822 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 620688 T1822 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 620689 T1822 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 620689 T1822 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 620689 T1822 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 620691 T1822 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 620691 T1822 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 620691 T1822 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 620691 T1822 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 620691 T1822 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-003/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-001/jetty1/
   [junit4]   2> 620692 T1822 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@706aca0d
   [junit4]   2> 620692 T1822 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-001/jetty1
   [junit4]   2> 620692 T1822 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-001/jetty1/index/
   [junit4]   2> 620692 T1822 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 620693 T1822 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-001/jetty1/index
   [junit4]   2> 620693 T1822 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1318373034, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 620694 T1822 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1c5baa04 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@530f3d70),segFN=segments_1,generation=1}
   [junit4]   2> 620694 T1822 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 620696 T1822 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 620697 T1822 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 620697 T1822 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 620697 T1822 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 620697 T1822 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 620697 T1822 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 620698 T1822 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 620698 T1822 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 620698 T1822 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 620699 T1822 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 620699 T1822 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 620700 T1822 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 620700 T1822 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 620701 T1822 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 620701 T1822 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 620701 T1822 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 620701 T1822 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 620702 T1822 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 620702 T1822 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 620703 T1822 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 620705 T1822 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 620706 T1822 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 620707 T1822 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 620708 T1822 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 620709 T1822 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 620710 T1822 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 620710 T1822 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=10, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 620711 T1822 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1c5baa04 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@530f3d70),segFN=segments_1,generation=1}
   [junit4]   2> 620711 T1822 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 620711 T1822 oass.SolrIndexSearcher.<init> Opening Searcher@7a1256ee[collection1] main
   [junit4]   2> 620712 T1822 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 620713 T1822 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 620713 T1822 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 620714 T1822 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 620714 T1822 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 620715 T1822 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 620716 T1822 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 620716 T1822 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 620716 T1822 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 620717 T1822 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 620721 T1823 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7a1256ee[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 620724 T1822 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 620725 T1826 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:48650/lm_wky/ic collection:collection1 shard:shard1
   [junit4]   2> 620726 T1773 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 620727 T1773 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 620727 T1826 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 620732 T1826 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 620733 T1800 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 620734 T1826 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 620734 T1826 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C749 name=collection1 org.apache.solr.core.SolrCore@2d681c15 url=http://127.0.0.1:48650/lm_wky/ic/collection1 node=127.0.0.1:48650_lm_wky%2Fic C749_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=http://127.0.0.1:48650/lm_wky/ic, node_name=127.0.0.1:48650_lm_wky%2Fic, state=down}
   [junit4]   2> 620734 T1826 C749 P48650 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:48650/lm_wky/ic/collection1/
   [junit4]   2> 620735 T1826 C749 P48650 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 620735 T1826 C749 P48650 oasc.SyncStrategy.syncToMe http://127.0.0.1:48650/lm_wky/ic/collection1/ has no replicas
   [junit4]   2> 620735 T1826 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:48650/lm_wky/ic/collection1/ shard1
   [junit4]   2> 620735 T1826 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 620738 T1800 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 620841 T1807 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 620841 T1800 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 620841 T1821 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 620889 T1826 oasc.ZkController.register We are http://127.0.0.1:48650/lm_wky/ic/collection1/ and leader is http://127.0.0.1:48650/lm_wky/ic/collection1/
   [junit4]   2> 620889 T1826 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:48650/lm_wky/ic
   [junit4]   2> 620889 T1826 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 620889 T1826 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 620889 T1826 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 620891 T1800 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 620892 T1796 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48650/lm_wky/ic",
   [junit4]   2> 	  "node_name":"127.0.0.1:48650_lm_wky%2Fic",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 620972 T1773 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 620973 T1773 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 620975 T1773 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:35483
   [junit4]   2> 620976 T1773 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 620976 T1773 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 620976 T1773 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-004
   [junit4]   2> 620976 T1773 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-004/'
   [junit4]   2> 620987 T1773 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-004/solr.xml
   [junit4]   2> 621005 T1807 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 621005 T1800 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 621005 T1821 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 621008 T1773 oasc.CoreContainer.<init> New CoreContainer 908440494
   [junit4]   2> 621008 T1773 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-004/]
   [junit4]   2> 621009 T1773 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 621009 T1773 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 621009 T1773 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 621009 T1773 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 621010 T1773 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 621010 T1773 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 621010 T1773 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 621011 T1773 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 621011 T1773 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 621011 T1773 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 621012 T1773 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 621012 T1773 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 621012 T1773 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 621013 T1773 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40048/solr
   [junit4]   2> 621013 T1773 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 621013 T1773 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 621014 T1773 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 621015 T1837 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@40d73089 name:ZooKeeperConnection Watcher:127.0.0.1:40048 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 621016 T1773 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 621016 T1773 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 621017 T1773 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 621018 T1839 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7a7b785e name:ZooKeeperConnection Watcher:127.0.0.1:40048/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 621018 T1773 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 621021 T1773 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 622024 T1773 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35483_lm_wky%2Fic
   [junit4]   2> 622024 T1773 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35483_lm_wky%2Fic
   [junit4]   2> 622027 T1773 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 622030 T1841 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 622031 T1841 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 622031 T1800 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 622031 T1841 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 622033 T1796 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:35483/lm_wky/ic",
   [junit4]   2> 	  "node_name":"127.0.0.1:35483_lm_wky%2Fic",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 622033 T1796 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
   [junit4]   2> 622033 T1796 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 622135 T1807 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 622135 T1800 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 622135 T1821 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 622135 T1840 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 623032 T1841 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 623032 T1841 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 623032 T1841 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 623033 T1841 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 623033 T1841 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 623033 T1841 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-004/collection1/'
   [junit4]   2> 623034 T1841 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 623034 T1841 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 623034 T1841 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 623052 T1841 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 623060 T1841 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 623061 T1841 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 623064 T1841 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 623153 T1841 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 623156 T1841 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 623158 T1841 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 623164 T1841 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 623166 T1841 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 623167 T1841 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 623168 T1841 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 623168 T1841 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 623169 T1841 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 623169 T1841 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 623169 T1841 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 623169 T1841 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 623169 T1841 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 623170 T1841 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-004/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-001/jetty2/
   [junit4]   2> 623170 T1841 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@706aca0d
   [junit4]   2> 623170 T1841 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-001/jetty2
   [junit4]   2> 623170 T1841 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-001/jetty2/index/
   [junit4]   2> 623171 T1841 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 623171 T1841 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-001/jetty2/index
   [junit4]   2> 623171 T1841 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1318373034, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 623172 T1841 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@28aaac7b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5ed130ce),segFN=segments_1,generation=1}
   [junit4]   2> 623172 T1841 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 623174 T1841 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 623175 T1841 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 623175 T1841 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 623175 T1841 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 623175 T1841 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 623175 T1841 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 623176 T1841 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 623176 T1841 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 623176 T1841 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 623176 T1841 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 623177 T1841 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 623177 T1841 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 623177 T1841 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 623177 T1841 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 623178 T1841 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 623178 T1841 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 623178 T1841 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 623178 T1841 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 623178 T1841 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 623179 T1841 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 623180 T1841 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 623181 T1841 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 623181 T1841 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 623182 T1841 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 623183 T1841 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 623183 T1841 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 623184 T1841 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=10, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 623184 T1841 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@28aaac7b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5ed130ce),segFN=segments_1,generation=1}
   [junit4]   2> 623184 T1841 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 623184 T1841 oass.SolrIndexSearcher.<init> Opening Searcher@7c9d20b1[collection1] main
   [junit4]   2> 623184 T1841 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 623185 T1841 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 623185 T1841 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 623185 T1841 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 623186 T1841 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 623186 T1841 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 623186 T1841 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 623186 T1841 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 623186 T1841 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 623187 T1841 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 623188 T1842 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7c9d20b1[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 623190 T1841 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 623191 T1845 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:35483/lm_wky/ic collection:collection1 shard:shard1
   [junit4]   2> 623191 T1773 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 623192 T1773 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 623193 T1845 oasc.ZkController.register We are http://127.0.0.1:35483/lm_wky/ic/collection1/ and leader is http://127.0.0.1:48650/lm_wky/ic/collection1/
   [junit4]   2> 623193 T1845 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:35483/lm_wky/ic
   [junit4]   2> 623193 T1845 oasc.ZkController.checkRecovery Core needs to recover:collection1
   [junit4]   2> 623193 T1845 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
   [junit4]   2> ASYNC  NEW_CORE C750 name=collection1 org.apache.solr.core.SolrCore@69a26c79 url=http://127.0.0.1:35483/lm_wky/ic/collection1 node=127.0.0.1:35483_lm_wky%2Fic C750_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=http://127.0.0.1:35483/lm_wky/ic, node_name=127.0.0.1:35483_lm_wky%2Fic, state=down}
   [junit4]   2> 623195 T1846 C750 P35483 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
   [junit4]   2> 623195 T1846 C750 P35483 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
   [junit4]   2> 623196 T1846 C750 P35483 oasc.RecoveryStrategy.doRecovery Publishing state of core collection1 as recovering, leader is http://127.0.0.1:48650/lm_wky/ic/collection1/ and I am http://127.0.0.1:35483/lm_wky/ic/collection1/
   [junit4]   2> 623196 T1846 C750 P35483 oasc.ZkController.publish publishing core=collection1 state=recovering collection=collection1
   [junit4]   2> 623196 T1846 C750 P35483 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 623197 T1800 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 623197 T1846 C750 P35483 oasc.RecoveryStrategy.sendPrepRecoveryCmd Sending prep recovery command to http://127.0.0.1:48650/lm_wky/ic; WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1%3A35483_lm_wky%252Fic&coreNodeName=core_node2&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true
   [junit4]   2> 623206 T1796 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:35483/lm_wky/ic",
   [junit4]   2> 	  "node_name":"127.0.0.1:35483_lm_wky%2Fic",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 623209 T1814 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
   [junit4]   2> 623211 T1814 oasha.CoreAdminHandler.handleWaitForStateAction Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
   [junit4]   2> 623211 T1814 oasha.CoreAdminHandler.handleWaitForStateAction In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:35483_lm_wky%2Fic, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","base_url":"http://127.0.0.1:35483/lm_wky/ic","node_name":"127.0.0.1:35483_lm_wky%2Fic","state":"down"}
   [junit4]   2> 623308 T1807 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 623308 T1800 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 623309 T1821 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 623313 T1840 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 623457 T1773 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 623457 T1773 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 623459 T1773 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:35313
   [junit4]   2> 623460 T1773 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 623460 T1773 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 623460 T1773 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-005
   [junit4]   2> 623460 T1773 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-005/'
   [junit4]   2> 623471 T1773 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-005/solr.xml
   [junit4]   2> 623488 T1773 oasc.CoreContainer.<init> New CoreContainer 1548773694
   [junit4]   2> 623489 T1773 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-005/]
   [junit4]   2> 623489 T1773 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 623489 T1773 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 623490 T1773 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 623490 T1773 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 623490 T1773 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 623490 T1773 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 623490 T1773 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 623491 T1773 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 623491 T1773 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 623491 T1773 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 623491 T1773 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 623492 T1773 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 623492 T1773 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 623492 T1773 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40048/solr
   [junit4]   2> 623492 T1773 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 623493 T1773 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 623493 T1773 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 623496 T1858 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@164724ce name:ZooKeeperConnection Watcher:127.0.0.1:40048 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 623497 T1773 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 623497 T1773 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 623498 T1773 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 623499 T1860 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@42cab3e4 name:ZooKeeperConnection Watcher:127.0.0.1:40048/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 623500 T1773 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 623503 T1773 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 624212 T1814 oasha.CoreAdminHandler.handleWaitForStateAction In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:35483_lm_wky%2Fic, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","base_url":"http://127.0.0.1:35483/lm_wky/ic","node_name":"127.0.0.1:35483_lm_wky%2Fic","state":"recovering"}
   [junit4]   2> 624212 T1814 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
   [junit4]   2> 624212 T1814 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:35483_lm_wky%252Fic&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node2&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1003 
   [junit4]   2> 624506 T1773 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35313_lm_wky%2Fic
   [junit4]   2> 624507 T1773 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35313_lm_wky%2Fic
   [junit4]   2> 624511 T1773 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 624513 T1862 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 624513 T1862 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 624514 T1800 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected ty

[...truncated too long message...]

de_name":"127.0.0.1:57024_lm_wky%2Fic",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 665725 T1922 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 665728 T1773 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=3,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=1,adds=1,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=5,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0,transaction_logs_total_size=487,transaction_logs_total_number=1}
   [junit4]   2> 665729 T1773 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 665729 T1773 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 665729 T1773 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2>  C761_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=http://127.0.0.1:57024/lm_wky/ic, node_name=127.0.0.1:57024_lm_wky%2Fic, state=down, leader=true}
   [junit4]   2> 665729 T1773 C761 P57024 oasu.DirectUpdateHandler2.closeWriter Committing on IndexWriter close.
   [junit4]   2> 665734 T1773 C761 P57024 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@525649a0 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@30f26746),segFN=segments_2,generation=2}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@525649a0 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@30f26746),segFN=segments_3,generation=3}
   [junit4]   2> 665734 T1773 C761 P57024 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
   [junit4]   2> 665735 T1773 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 665736 T1773 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 665736 T1773 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-001/jetty4 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-001/jetty4;done=false>>]
   [junit4]   2> 665736 T1773 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-001/jetty4
   [junit4]   2> 665737 T1773 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-001/jetty4/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-001/jetty4/index;done=false>>]
   [junit4]   2> 665737 T1773 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001/tempDir-001/jetty4/index
   [junit4]   2> 665738 T1773 oasc.Overseer.close Overseer (id=92442518116171788-127.0.0.1:57024_lm_wky%2Fic-n_0000000004) closing
   [junit4]   2> 665739 T1923 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:57024_lm_wky%2Fic
   [junit4]   2> 665740 T1922 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 665762 T1773 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/lm_wky/ic,null}
   [junit4]   2> 667240 T1926 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:347)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:344)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:344)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:346)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:296)
   [junit4]   2> 
   [junit4]   2> 670752 T1773 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 670753 T1773 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:40048 40048
   [junit4]   2> 671569 T1774 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:40048 40048
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SyncSliceTest -Dtests.method=testDistribSearch -Dtests.seed=E071A2599265EA36 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=mt_MT -Dtests.timezone=CST -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 55.1s J0 | SyncSliceTest.testDistribSearch <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: expected:<5> but was:<4>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E071A2599265EA36:61972C41E53A8A0A]:0)
   [junit4]    > 	at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:176)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 671604 T1773 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.SyncSliceTest-E071A2599265EA36-001
   [junit4]   2> NOTE: test params are: codec=Lucene410: {rnd_b=PostingsFormat(name=Memory doPackFST= true), other_tl1=FSTOrd41, _version_=FSTOrd41, multiDefault=PostingsFormat(name=MockRandom), a_t=FSTOrd41, intDefault=FSTOrd41, a_i1=FSTOrd41, id=PostingsFormat(name=Memory doPackFST= true), text=PostingsFormat(name=Lucene41WithOrds), range_facet_l=FSTOrd41, timestamp=FSTOrd41}, docValues:{timestamp=DocValuesFormat(name=Lucene410)}, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=mt_MT, timezone=CST
   [junit4]   2> NOTE: Linux 3.13.0-35-generic amd64/Oracle Corporation 1.8.0_20 (64-bit)/cpus=8,threads=1,free=40085480,total=194031616
   [junit4]   2> NOTE: All tests run in this JVM: [TestNonNRTOpen, TestDynamicFieldResource, PeerSyncTest, LeaderElectionTest, BlockCacheTest, RequiredFieldsTest, TestSolrDeletionPolicy2, OutOfBoxZkACLAndCredentialsProvidersTest, DisMaxRequestHandlerTest, RollingRestartTest, TestFastWriter, TestIntervalFaceting, MinimalSchemaTest, TestAnalyzedSuggestions, DistributedQueryComponentCustomSortTest, FileUtilsTest, HighlighterMaxOffsetTest, TestRandomDVFaceting, ExpressionTest, SuggesterTSTTest, TestOrdValues, AddSchemaFieldsUpdateProcessorFactoryTest, DistributedSuggestComponentTest, TestFastLRUCache, ResponseLogComponentTest, TestConfigSets, ShardRoutingTest, TestSolrXmlPersistence, TestFastOutputStream, TestArbitraryIndexDir, ZkControllerTest, SpellCheckComponentTest, FastVectorHighlighterTest, TestHashPartitioner, PreAnalyzedUpdateProcessorTest, SystemInfoHandlerTest, StandardRequestHandlerTest, SignatureUpdateProcessorFactoryTest, ExternalCollectionsTest, ChangedSchemaMergeTest, MBeansHandlerTest, SolrIndexSplitterTest, DeleteReplicaTest, ExternalFileFieldSortTest, TestFuzzyAnalyzedSuggestions, EchoParamsTest, DocumentBuilderTest, SyncSliceTest]
   [junit4] Completed on J0 in 55.13s, 1 test, 1 failure <<< FAILURES!

[...truncated 1028 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:471: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:60: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1358: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:961: There were test failures: 438 suites, 1830 tests, 1 failure, 45 ignored (23 assumptions)

Total time: 96 minutes 24 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_20 -XX:+UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any