You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/03/19 04:53:15 UTC

[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_60-ea-b10) - Build # 9841 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9841/
Java: 32bit/jdk1.7.0_60-ea-b10 -client -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.testWithin {#9 seed=[14A074E14CFDBEA3:F66F35851492BBDC]}

Error Message:
Shouldn't match I#3:Rect(minX=-166.0,maxX=-161.0,minY=87.0,maxY=90.0) Q:Pt(x=180.0,y=56.0)

Stack Trace:
java.lang.AssertionError: Shouldn't match I#3:Rect(minX=-166.0,maxX=-161.0,minY=87.0,maxY=90.0) Q:Pt(x=180.0,y=56.0)
	at __randomizedtesting.SeedInfo.seed([14A074E14CFDBEA3:F66F35851492BBDC]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.fail(SpatialOpRecursivePrefixTreeTest.java:355)
	at org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.doTest(SpatialOpRecursivePrefixTreeTest.java:335)
	at org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.testWithin(SpatialOpRecursivePrefixTreeTest.java:119)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:826)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:862)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 9265 lines...]
   [junit4] Suite: org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(GeohashPrefixTree(maxLevels:1,ctx:SpatialContext.GEO)))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxLevels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(maxLevels:3,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(GeohashPrefixTree(maxLevels:2,ctx:SpatialContext.GEO)))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxLevels:4,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(GeohashPrefixTree(maxLevels:1,ctx:SpatialContext.GEO)))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(GeohashPrefixTree(maxLevels:1,ctx:SpatialContext.GEO)))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(maxLevels:3,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxLevels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(GeohashPrefixTree(maxLevels:1,ctx:SpatialContext.GEO)))
   [junit4]   2> Ig:Rect(minX=-180.0,maxX=-135.0,minY=45.0,maxY=90.0) Qg:Rect(minX=135.0,maxX=180.0,minY=45.0,maxY=90.0)
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SpatialOpRecursivePrefixTreeTest -Dtests.method=testWithin -Dtests.seed=14A074E14CFDBEA3 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=sv -Dtests.timezone=Pacific/Kosrae -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 0.13s J0 | SpatialOpRecursivePrefixTreeTest.testWithin {#9 seed=[14A074E14CFDBEA3:F66F35851492BBDC]} <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Shouldn't match I#3:Rect(minX=-166.0,maxX=-161.0,minY=87.0,maxY=90.0) Q:Pt(x=180.0,y=56.0)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([14A074E14CFDBEA3:F66F35851492BBDC]:0)
   [junit4]    > 	at org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.fail(SpatialOpRecursivePrefixTreeTest.java:355)
   [junit4]    > 	at org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.doTest(SpatialOpRecursivePrefixTreeTest.java:335)
   [junit4]    > 	at org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.testWithin(SpatialOpRecursivePrefixTreeTest.java:119)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(GeohashPrefixTree(maxLevels:2,ctx:SpatialContext.GEO)))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(GeohashPrefixTree(maxLevels:3,ctx:SpatialContext.GEO)))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxLevels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxLevels:4,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxLevels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxLevels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxLevels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(GeohashPrefixTree(maxLevels:2,ctx:SpatialContext.GEO)))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(GeohashPrefixTree(maxLevels:3,ctx:SpatialContext.GEO)))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:1,SPG:(QuadPrefixTree(maxLevels:5,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(GeohashPrefixTree(maxLevels:3,ctx:SpatialContext.GEO)))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:1,SPG:(QuadPrefixTree(maxLevels:5,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxLevels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(GeohashPrefixTree(maxLevels:2,ctx:SpatialContext.GEO)))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxLevels:4,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(maxLevels:3,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxLevels:4,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(GeohashPrefixTree(maxLevels:2,ctx:SpatialContext.GEO)))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(GeohashPrefixTree(maxLevels:3,ctx:SpatialContext.GEO)))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxLevels:4,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxLevels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(maxLevels:3,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(maxLevels:3,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
   [junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(GeohashPrefixTree(maxLevels:3,ctx:SpatialContext.GEO)))
   [junit4]   2> NOTE: test params are: codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=225), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=225)), sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=sv, timezone=Pacific/Kosrae
   [junit4]   2> NOTE: Linux 3.8.0-37-generic i386/Oracle Corporation 1.7.0_60-ea (32-bit)/cpus=8,threads=1,free=16393832,total=91226112
   [junit4]   2> NOTE: All tests run in this JVM: [SpatialPrefixTreeTest, TestRecursivePrefixTreeStrategy, SpatialOpRecursivePrefixTreeTest]
   [junit4] Completed on J0 in 1.99s, 43 tests, 1 failure <<< FAILURES!

[...truncated 35 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build.xml:539: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1996: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/module-build.xml:60: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1275: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 14 suites, 79 tests, 1 failure, 2 ignored (2 assumptions)

Total time: 32 minutes 13 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_60-ea-b10 -client -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_51) - Build # 9851 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9851/
Java: 64bit/jdk1.7.0_51 -XX:-UseCompressedOops -XX:+UseParallelGC -XX:-UseSuperWord

1 tests failed.
REGRESSION:  org.apache.solr.client.solrj.impl.CloudSolrServerTest.testDistribSearch

Error Message:
java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:58193 within 45000 ms

Stack Trace:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:58193 within 45000 ms
	at __randomizedtesting.SeedInfo.seed([DA14EE190BDB3D4B:5BF260017C845D77]:0)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:91)
	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:91)
	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:85)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:75)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:201)
	at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:78)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:860)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:58193 within 45000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
	... 45 more




Build Log:
[...truncated 11692 lines...]
   [junit4] Suite: org.apache.solr.client.solrj.impl.CloudSolrServerTest
   [junit4]   2> 12228 T55 oas.SolrTestCaseJ4.startTrackingSearchers WARN startTrackingSearchers: numOpens=6 numCloses=6
   [junit4]   2> 12229 T55 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
   [junit4]   2> 12229 T55 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 12231 T55 oasc.AbstractZkTestCase.<clinit> WARN TEST_HOME() does not exist - solrj test?
   [junit4]   2> 12240 T55 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1395277412196
   [junit4]   2> 12242 T55 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 12244 T56 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 12344 T55 oasc.ZkTestServer.run start zk server on port:58193
   [junit4]   2> 12385 T55 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 77224 T59 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 64820ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
   [junit4]   2> 77225 T55 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 77244 T57 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x144dd03b79e0000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 77247 T57 oazs.NIOServerCnxnFactory.run WARN Ignoring unexpected runtime exception java.nio.channels.CancelledKeyException
   [junit4]   2> 	at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
   [junit4]   2> 	at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:187)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 77251 T55 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:58193 58193
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CloudSolrServerTest -Dtests.method=testDistribSearch -Dtests.seed=DA14EE190BDB3D4B -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_MA -Dtests.timezone=America/Creston -Dtests.file.encoding=UTF-8
   [junit4] ERROR   85.2s J0 | CloudSolrServerTest.testDistribSearch <<<
   [junit4]    > Throwable #1: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:58193 within 45000 ms
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DA14EE190BDB3D4B:5BF260017C845D77]:0)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:91)
   [junit4]    > 	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:91)
   [junit4]    > 	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:85)
   [junit4]    > 	at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:75)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:201)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:78)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]    > Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:58193 within 45000 ms
   [junit4]    > 	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
   [junit4]    > 	... 45 more
   [junit4]   2> 97465 T55 oas.SolrTestCaseJ4.setUp ###Starting testShutdown
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1395277497422
   [junit4]   2> 97466 T55 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 97467 T63 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 97567 T55 oasc.ZkTestServer.run start zk server on port:58111
   [junit4]   2> 97569 T55 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 97576 T69 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7291279 name:ZooKeeperConnection Watcher:127.0.0.1:58111 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 97578 T55 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 97581 T55 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 97629 T55 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 97633 T71 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@335dd1d3 name:ZooKeeperConnection Watcher:127.0.0.1:58111/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 97634 T55 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 97644 T55 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 97653 T55 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 97658 T55 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 97662 T55 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 97668 T55 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 97675 T55 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 97682 T55 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 97682 T55 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 97687 T55 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml because it doesn't exist
   [junit4]   2> 97687 T55 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/stopwords.txt because it doesn't exist
   [junit4]   2> 97688 T55 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/protwords.txt because it doesn't exist
   [junit4]   2> 97688 T55 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/currency.xml because it doesn't exist
   [junit4]   2> 97689 T55 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/enumsConfig.xml because it doesn't exist
   [junit4]   2> 97689 T55 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/open-exchange-rates.json because it doesn't exist
   [junit4]   2> 97689 T55 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/mapping-ISOLatin1Accent.txt because it doesn't exist
   [junit4]   2> 97690 T55 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/old_synonyms.txt because it doesn't exist
   [junit4]   2> 97690 T55 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/synonyms.txt because it doesn't exist
   [junit4]   2> 97698 T55 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 97700 T72 oaz.ClientCnxnSocketNIO.connect ERROR Unable to open socket to ff01::114/ff01:0:0:0:0:0:0:114:33332
   [junit4]   2> 97700 T72 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.SocketException: Network is unreachable
   [junit4]   2> 	at sun.nio.ch.Net.connect0(Native Method)
   [junit4]   2> 	at sun.nio.ch.Net.connect(Net.java:465)
   [junit4]   2> 	at sun.nio.ch.Net.connect(Net.java:457)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:277)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:287)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:967)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]   2> 
   [junit4]   2> 98803 T72 oaz.ClientCnxnSocketNIO.connect ERROR Unable to open socket to ff01::114/ff01:0:0:0:0:0:0:114:33332
   [junit4]   2> 98906 T55 oas.SolrTestCaseJ4.tearDown ###Ending testShutdown
   [junit4]   2> 98908 T55 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:58111 58111
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 118975 T55 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=ar_MA, timezone=America/Creston
   [junit4]   2> NOTE: Linux 3.8.0-37-generic amd64/Oracle Corporation 1.7.0_51 (64-bit)/cpus=8,threads=1,free=108076632,total=260046848
   [junit4]   2> NOTE: All tests run in this JVM: [SolrParamTest, TestDocumentObjectBinder, LargeVolumeJettyTest, TestEmbeddedSolrServer, CloudSolrServerTest]
   [junit4] Completed on J0 in 106.81s, 2 tests, 1 error <<< FAILURES!

[...truncated 103 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:202: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1275: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 49 suites, 283 tests, 1 error

Total time: 66 minutes 28 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_51 -XX:-UseCompressedOops -XX:+UseParallelGC -XX:-UseSuperWord
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_51) - Build # 9850 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9850/
Java: 64bit/jdk1.7.0_51 -XX:-UseCompressedOops -XX:+UseParallelGC -XX:-UseSuperWord

All tests passed

Build Log:
[...truncated 43679 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:63: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build.xml:208: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build.xml:543: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:2342: java.net.UnknownHostException: issues.apache.org
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:178)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:579)
	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618)
	at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:160)
	at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
	at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:275)
	at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:371)
	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932)
	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
	at org.apache.tools.ant.taskdefs.Get$GetThread.openConnection(Get.java:660)
	at org.apache.tools.ant.taskdefs.Get$GetThread.get(Get.java:579)
	at org.apache.tools.ant.taskdefs.Get$GetThread.run(Get.java:569)

Total time: 78 minutes 42 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_51 -XX:-UseCompressedOops -XX:+UseParallelGC -XX:-UseSuperWord
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_51) - Build # 9849 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9849/
Java: 64bit/jdk1.7.0_51 -XX:-UseCompressedOops -XX:+UseSerialGC -XX:-UseSuperWord

1 tests failed.
FAILED:  org.apache.solr.client.solrj.impl.CloudSolrServerTest.testShutdown

Error Message:
java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:47996 within 45000 ms

Stack Trace:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:47996 within 45000 ms
	at __randomizedtesting.SeedInfo.seed([1E59FC2A1DB0BBA5:FD2FF5BFFACA2ED7]:0)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:91)
	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:91)
	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:85)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:75)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:201)
	at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:78)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:860)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:47996 within 45000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
	... 45 more




Build Log:
[...truncated 128 lines...]
ERROR: Failed to update http://svn.apache.org/repos/asf/lucene/dev/trunk
org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS /repos/asf/lucene/dev/trunk failed
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:388)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:373)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:361)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:707)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:627)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:102)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1020)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:148)
	at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:339)
	at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:328)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:482)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:364)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:274)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:27)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:11)
	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1238)
	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:157)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1029)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1010)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:986)
	at hudson.FilePath.act(FilePath.java:914)
	at hudson.FilePath.act(FilePath.java:887)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:935)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:870)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1414)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:671)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:580)
	at hudson.model.Run.execute(Run.java:1676)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:231)
Caused by: svn: E175002: OPTIONS /repos/asf/lucene/dev/trunk failed
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:154)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:97)
	... 38 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/trunk'
svn: E175002: unknown host
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:777)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:382)
	... 37 more
Caused by: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/trunk'
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:775)
	... 38 more
Caused by: svn: E175002: unknown host
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:517)
	... 38 more
Caused by: java.net.UnknownHostException: svn.apache.org: Name or service not known
	at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
	at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901)
	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1293)
	at java.net.InetAddress.getAllByName0(InetAddress.java:1246)
	at java.net.InetAddress.getAllByName(InetAddress.java:1162)
	at java.net.InetAddress.getAllByName(InetAddress.java:1098)
	at java.net.InetAddress.getByName(InetAddress.java:1048)
	at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.createAddres(SVNSocketFactory.java:189)
	at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.createPlainSocket(SVNSocketFactory.java:66)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.connect(HTTPConnection.java:280)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:451)
	... 38 more
ERROR: Subversion update failed
java.io.IOException
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:207)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1029)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1010)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:986)
	at hudson.FilePath.act(FilePath.java:914)
	at hudson.FilePath.act(FilePath.java:887)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:935)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:870)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1414)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:671)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:580)
	at hudson.model.Run.execute(Run.java:1676)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:231)
Caused by: hudson.scm.subversion.UpdaterException: failed to perform svn update
	... 17 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS /repos/asf/lucene/dev/trunk failed
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:388)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:373)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:361)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:707)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:627)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:102)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1020)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:148)
	at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:339)
	at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:328)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:482)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:364)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:274)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:27)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:11)
	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1238)
	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:157)
	... 16 more
Caused by: svn: E175002: OPTIONS /repos/asf/lucene/dev/trunk failed
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:154)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:97)
	... 38 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/trunk'
svn: E175002: unknown host
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:777)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:382)
	... 37 more
Caused by: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/trunk'
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:775)
	... 38 more
Caused by: svn: E175002: unknown host
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:517)
	... 38 more
Caused by: java.net.UnknownHostException: svn.apache.org: Name or service not known
	at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
	at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901)
	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1293)
	at java.net.InetAddress.getAllByName0(InetAddress.java:1246)
	at java.net.InetAddress.getAllByName(InetAddress.java:1162)
	at java.net.InetAddress.getAllByName(InetAddress.java:1098)
	at java.net.InetAddress.getByName(InetAddress.java:1048)
	at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.createAddres(SVNSocketFactory.java:189)
	at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.createPlainSocket(SVNSocketFactory.java:66)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.connect(HTTPConnection.java:280)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:451)
	... 38 more
Retrying after 10 seconds
Cleaning up /var/lib/jenkins/workspace/Lucene-Solr-trunk-Linux/.
Updating http://svn.apache.org/repos/asf/lucene/dev/trunk at revision '2014-03-19T19:10:59.373 +0000'
ERROR: Failed to update http://svn.apache.org/repos/asf/lucene/dev/trunk
org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS /repos/asf/lucene/dev/trunk failed
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:388)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:373)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:361)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:707)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:627)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:102)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1020)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:148)
	at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:339)
	at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:328)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:482)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:364)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:274)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:27)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:11)
	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1238)
	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:157)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1029)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1010)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:986)
	at hudson.FilePath.act(FilePath.java:914)
	at hudson.FilePath.act(FilePath.java:887)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:935)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:870)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1414)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:671)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:580)
	at hudson.model.Run.execute(Run.java:1676)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:231)
Caused by: svn: E175002: OPTIONS /repos/asf/lucene/dev/trunk failed
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:154)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:97)
	... 38 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/trunk'
svn: E175002: unknown host
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:777)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:382)
	... 37 more
Caused by: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/trunk'
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:775)
	... 38 more
Caused by: svn: E175002: unknown host
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:517)
	... 38 more
Caused by: java.net.UnknownHostException: svn.apache.org
	at java.net.InetAddress.getAllByName0(InetAddress.java:1250)
	at java.net.InetAddress.getAllByName(InetAddress.java:1162)
	at java.net.InetAddress.getAllByName(InetAddress.java:1098)
	at java.net.InetAddress.getByName(InetAddress.java:1048)
	at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.createAddres(SVNSocketFactory.java:189)
	at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.createPlainSocket(SVNSocketFactory.java:66)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.connect(HTTPConnection.java:280)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:451)
	... 38 more
ERROR: Subversion update failed
java.io.IOException
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:207)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1029)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1010)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:986)
	at hudson.FilePath.act(FilePath.java:914)
	at hudson.FilePath.act(FilePath.java:887)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:935)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:870)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1414)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:671)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:580)
	at hudson.model.Run.execute(Run.java:1676)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:231)
Caused by: hudson.scm.subversion.UpdaterException: failed to perform svn update
	... 17 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS /repos/asf/lucene/dev/trunk failed
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:388)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:373)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:361)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:707)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:627)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:102)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1020)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:148)
	at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:339)
	at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:328)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:482)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:364)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:274)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:27)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:11)
	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1238)
	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:157)
	... 16 more
Caused by: svn: E175002: OPTIONS /repos/asf/lucene/dev/trunk failed
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:154)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:97)
	... 38 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/trunk'
svn: E175002: unknown host
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:777)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:382)
	... 37 more
Caused by: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/trunk'
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:775)
	... 38 more
Caused by: svn: E175002: unknown host
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:517)
	... 38 more
Caused by: java.net.UnknownHostException: svn.apache.org
	at java.net.InetAddress.getAllByName0(InetAddress.java:1250)
	at java.net.InetAddress.getAllByName(InetAddress.java:1162)
	at java.net.InetAddress.getAllByName(InetAddress.java:1098)
	at java.net.InetAddress.getByName(InetAddress.java:1048)
	at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.createAddres(SVNSocketFactory.java:189)
	at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.createPlainSocket(SVNSocketFactory.java:66)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.connect(HTTPConnection.java:280)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:451)
	... 38 more
Retrying after 10 seconds
Cleaning up /var/lib/jenkins/workspace/Lucene-Solr-trunk-Linux/.
Updating http://svn.apache.org/repos/asf/lucene/dev/trunk at revision '2014-03-19T19:10:59.373 +0000'
U         lucene/ivy-versions.properties
G         solr/CHANGES.txt
U         solr/core/src/test/org/apache/solr/cloud/DeleteShardTest.java
U         solr/core/src/test/org/apache/solr/cloud/DeleteInactiveReplicaTest.java
U         solr/core/src/test/org/apache/solr/cloud/SyncSliceTest.java
U         solr/core/src/test/org/apache/solr/cloud/AliasIntegrationTest.java
U         solr/core/src/test/org/apache/solr/cloud/ShardSplitTest.java
U         solr/core/src/test/org/apache/solr/cloud/MigrateRouteKeyTest.java
U         solr/core/src/test/org/apache/solr/cloud/CollectionsAPIDistributedZkTest.java
U         solr/core/src/test/org/apache/solr/cloud/CustomCollectionTest.java
U         solr/core/src/test/org/apache/solr/cloud/UnloadDistributedZkTest.java
U         solr/core/src/test/org/apache/solr/cloud/ChaosMonkeyNothingIsSafeTest.java
U         solr/core/src/test/org/apache/solr/core/OpenCloseCoreStressTest.java
U         solr/core/src/test/org/apache/solr/handler/TestReplicationHandler.java
U         solr/solrj/src/test/org/apache/solr/client/solrj/embedded/MultiCoreEmbeddedTest.java
U         solr/solrj/src/test/org/apache/solr/client/solrj/embedded/MultiCoreExampleJettyTest.java
U         solr/solrj/src/test/org/apache/solr/client/solrj/MultiCoreExampleTestBase.java
U         solr/solrj/src/test/org/apache/solr/client/solrj/LargeVolumeTestBase.java
U         solr/solrj/src/test/org/apache/solr/client/solrj/response/NoOpResponseParserTest.java
U         solr/solrj/src/test/org/apache/solr/client/solrj/SolrExampleTests.java
U         solr/contrib/dataimporthandler/src/test/org/apache/solr/handler/dataimport/TestContentStreamDataSource.java
U         solr/contrib/map-reduce/src/test/org/apache/solr/hadoop/MorphlineGoLiveMiniMRTest.java
U         solr/contrib/map-reduce/src/java/org/apache/solr/hadoop/ZooKeeperInspector.java
U         solr/contrib/map-reduce/src/java/org/apache/solr/hadoop/SolrRecordWriter.java
U         solr/contrib/map-reduce/src/java/org/apache/solr/hadoop/MapReduceIndexerTool.java
U         solr/contrib/morphlines-core/src/test/org/apache/solr/morphlines/solr/AbstractSolrMorphlineTestBase.java
D         solr/licenses/zookeeper-3.4.5.jar.sha1
AU        solr/licenses/zookeeper-3.4.6.jar.sha1
U         solr/test-framework/src/java/org/apache/solr/cloud/AbstractFullDistribZkTestBase.java
U         solr/test-framework/src/java/org/apache/solr/SolrJettyTestBase.java
At revision 1579357
No emails were triggered.
[Lucene-Solr-trunk-Linux] $ /bin/sh -xe /tmp/hudson5151371078360309682.sh
+ echo Using JDK: 64bit/jdk1.7.0_51 -XX:-UseCompressedOops -XX:+UseSerialGC -XX:-UseSuperWord
Using JDK: 64bit/jdk1.7.0_51 -XX:-UseCompressedOops -XX:+UseSerialGC -XX:-UseSuperWord
+ /var/lib/jenkins/tools/java/64bit/jdk1.7.0_51/bin/java -XX:-UseCompressedOops -XX:+UseSerialGC -XX:-UseSuperWord -version
java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)
[Lucene-Solr-trunk-Linux] $ /var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/bin/ant "-Dargs=-XX:-UseCompressedOops -XX:+UseSerialGC -XX:-UseSuperWord" -Dtests.disableHdfs=true -Dtests.multiplier=3 -Dtests.jvms=2 jenkins-hourly
Buildfile: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml

jenkins-hourly:

clean:

clean:
     [echo] Building solr...

clean:

-test-with-heapdumps-enabled:
     [echo] Java HotSpot(TM) 64-Bit Server VM: Enabling heap dumps on OutOfMemoryError to dir '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/heapdumps'.
    [mkdir] Created dir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/heapdumps

test:

-run-test:
    [mkdir] Created dir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build

ivy-availability-check:

ivy-fail:

ivy-configure:
[ivy:configure] :: Apache Ivy 2.3.0 - 20130110142753 :: http://ant.apache.org/ivy/ ::
[ivy:configure] :: loading settings :: file = /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/ivy-settings.xml

-clover.load:

resolve-groovy:
[ivy:cachepath] :: resolving dependencies :: org.codehaus.groovy#groovy-all-caller;working
[ivy:cachepath] 	confs: [default]
[ivy:cachepath] 	found org.codehaus.groovy#groovy-all;2.2.1 in public
[ivy:cachepath] :: resolution report :: resolve 157ms :: artifacts dl 3ms
	---------------------------------------------------------------------
	|                  |            modules            ||   artifacts   |
	|       conf       | number| search|dwnlded|evicted|| number|dwnlded|
	---------------------------------------------------------------------
	|      default     |   1   |   0   |   0   |   0   ||   1   |   0   |
	---------------------------------------------------------------------

-init-totals:

test-core:

-clover.disable:

ivy-availability-check:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/ivy-settings.xml

-clover.load:

-clover.classpath:

-clover.setup:

clover:

resolve:

init:

compile-core:
    [mkdir] Created dir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build/core/classes/java
    [javac] Compiling 690 source files to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build/core/classes/java
    [javac] Note: Some input files use or override a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
     [copy] Copying 3 files to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build/core/classes/java

compile-test-framework:

ivy-availability-check:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/ivy-settings.xml

resolve:

init:

compile-lucene-core:

ivy-availability-check:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/ivy-settings.xml

resolve:

init:

-clover.disable:

-clover.load:

-clover.classpath:

-clover.setup:

clover:

compile-core:

compile-codecs:
     [echo] Building codecs...

ivy-availability-check:
     [echo] Building codecs...

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/ivy-settings.xml

resolve:

common.init:

compile-lucene-core:

init:

-clover.disable:

-clover.load:

-clover.classpath:

-clover.setup:

clover:

compile-core:
    [mkdir] Created dir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build/codecs/classes/java
    [javac] Compiling 71 source files to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build/codecs/classes/java
     [copy] Copying 3 files to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build/codecs/classes/java

-clover.disable:

-clover.load:

-clover.classpath:

-clover.setup:

clover:

common.compile-core:
    [mkdir] Created dir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build/test-framework/classes/java
    [javac] Compiling 130 source files to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build/test-framework/classes/java
    [javac] Note: Some input files use or override a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
     [copy] Copying 4 files to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build/test-framework/classes/java

compile-core:

compile-test:
    [mkdir] Created dir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build/core/classes/test
    [javac] Compiling 398 source files to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build/core/classes/test
    [javac] Note: Some input files use or override a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
     [copy] Copying 38 files to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build/core/classes/test

install-junit4-taskdef:

validate:

resolve-groovy:

-init-totals:

-test:
    [mkdir] Created dir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build/core/test
    [mkdir] Created dir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build/core/test/temp
    [mkdir] Created dir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/.caches/test-stats/core
   [junit4] <JUnit4> says jolly good day! Master seed: 2E6C351CC990CBA5
   [junit4] Executing 382 suites with 2 JVMs.
   [junit4] 
   [junit4] Started J0 PID(27938@serv1.sd-datasolutions.de).
   [junit4] Started J1 PID(27937@serv1.sd-datasolutions.de).
   [junit4] Suite: org.apache.lucene.search.TestBooleanQueryVisitSubscorers
   [junit4] Completed on J1 in 1.33s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestTermScorer
   [junit4] Completed on J1 in 0.24s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestMinShouldMatch2
   [junit4] Completed on J1 in 5.32s, 6 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestFlushByRamOrCountsPolicy
   [junit4] Completed on J0 in 7.15s, 5 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestTermVectorsReader
   [junit4] Completed on J1 in 0.36s, 6 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestTransactions
   [junit4] Completed on J1 in 2.00s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestNamedSPILoader
   [junit4] Completed on J1 in 0.13s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestDoubleBarrelLRUCache
   [junit4] Completed on J1 in 1.24s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestSearcherManager
   [junit4] Completed on J0 in 8.66s, 8 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestSearchWithThreads
   [junit4] Completed on J1 in 5.11s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterNRTIsCurrent
   [junit4] Completed on J1 in 5.19s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestSloppyPhraseQuery
   [junit4] Completed on J0 in 5.40s, 9 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestPayloads
   [junit4] Completed on J0 in 0.51s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestLazyProxSkipping
   [junit4] Completed on J1 in 0.77s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestIdentityHashSet
   [junit4] Completed on J1 in 0.23s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.analysis.TestLookaheadTokenFilter
   [junit4] Completed on J0 in 9.46s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestDirectoryReader
   [junit4] Completed on J0 in 1.60s, 25 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestCrashCausesCorruptIndex
   [junit4] Completed on J0 in 0.30s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestSameScoresWithThreads
   [junit4] Completed on J0 in 1.45s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestConcurrentMergeScheduler
   [junit4] Completed on J0 in 7.74s, 6 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestLongBitSet
   [junit4] Completed on J0 in 8.53s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestParallelTermEnum
   [junit4] Completed on J0 in 0.03s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestCachingCollector
   [junit4] Completed on J0 in 0.04s, 5 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestExceedMaxTermLength
   [junit4] Completed on J0 in 0.08s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.codecs.compressing.TestCompressingStoredFieldsFormat
   [junit4] IGNOR/A 0.24s J1 | TestCompressingStoredFieldsFormat.testBigDocuments {#0 seed=[2E6C351CC990CBA5:DF466F437ABC6582]}
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly())
   [junit4] IGNOR/A 0.00s J1 | TestCompressingStoredFieldsFormat.testBigDocuments {#1 seed=[2E6C351CC990CBA5:6B10D3BF4E7EAEAE]}
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly())
   [junit4] IGNOR/A 0.00s J1 | TestCompressingStoredFieldsFormat.testBigDocuments {#2 seed=[2E6C351CC990CBA5:E5F945631FBAE665]}
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly())
   [junit4] IGNOR/A 0.00s J1 | TestCompressingStoredFieldsFormat.testBigDocuments {#3 seed=[2E6C351CC990CBA5:D417EE867344BD4C]}
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly())
   [junit4] IGNOR/A 0.00s J1 | TestCompressingStoredFieldsFormat.testBigDocuments {#4 seed=[2E6C351CC990CBA5:98D66B2BD24C7DF7]}
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly())
   [junit4] Completed on J1 in 29.48s, 55 tests, 5 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestCodecHoldsOpenFiles
   [junit4] Completed on J1 in 0.09s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestFieldsReader
   [junit4] Completed on J0 in 0.72s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.store.TestBufferedIndexInput
   [junit4] Completed on J1 in 0.33s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestFlex
   [junit4] Completed on J1 in 0.11s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIntBlockPool
   [junit4] Completed on J1 in 0.85s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestDocTermOrdsRewriteMethod
   [junit4] Completed on J0 in 4.75s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestMultiDocValues
   [junit4] Completed on J1 in 3.77s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.junitcompat.TestSetupTeardownChaining
   [junit4] Completed on J1 in 0.13s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestDocTermOrds
   [junit4] Completed on J0 in 1.44s, 5 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.codecs.lucene40.TestReuseDocsEnum
   [junit4] Completed on J0 in 1.29s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.junitcompat.TestFailIfDirectoryNotClosed
   [junit4] Completed on J0 in 0.05s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexCommit
   [junit4] Completed on J0 in 0.02s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.spans.TestBasics
   [junit4] Completed on J1 in 4.67s, 28 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestSloppyMath
   [junit4] Completed on J1 in 0.16s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestControlledRealTimeReopenThread
   [junit4] Completed on J0 in 10.88s, 5 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestWildcard
   [junit4] Completed on J0 in 0.22s, 8 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestMultiPhraseQuery
   [junit4] Completed on J0 in 0.58s, 17 tests, 1 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestBoolean2
   [junit4] Completed on J1 in 9.56s, 11 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestCheckIndex
   [junit4] Completed on J1 in 0.11s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestTermVectorsWriter
   [junit4] Completed on J0 in 0.36s, 13 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestBooleanScorer
   [junit4] Completed on J1 in 0.10s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.junitcompat.TestReproduceMessage
   [junit4] Completed on J0 in 0.50s, 21 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestDocValuesWithThreads
   [junit4] Completed on J0 in 0.19s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestDocIdSet
   [junit4] Completed on J0 in 0.12s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestPhrasePrefixQuery
   [junit4] Completed on J0 in 0.03s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestMultiThreadTermVectors
   [junit4] Completed on J1 in 1.48s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestDateFilter
   [junit4] Completed on J1 in 0.07s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.spans.TestPayloadSpans
   [junit4] Completed on J1 in 0.92s, 10 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestBagOfPostings
   [junit4] Completed on J1 in 3.11s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestForceMergeForever
   [junit4] Completed on J1 in 0.61s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestTieredMergePolicy
   [junit4] Completed on J1 in 5.38s, 5 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.codecs.lucene40.TestLucene40TermVectorsFormat
   [junit4] Completed on J1 in 10.33s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.codecs.lucene42.TestLucene42DocValuesFormat
   [junit4] IGNOR/A 0.04s J0 | TestLucene42DocValuesFormat.testByteMissingVsFieldCache
   [junit4]    > Assumption #1: Codec does not support getDocsWithField
   [junit4] IGNOR/A 0.01s J0 | TestLucene42DocValuesFormat.testIntMissingVsFieldCache
   [junit4]    > Assumption #1: Codec does not support getDocsWithField
   [junit4] IGNOR/A 0.01s J0 | TestLucene42DocValuesFormat.testThreads2
   [junit4]    > Assumption #1: Codec does not support getDocsWithField
   [junit4] IGNOR/A 0.01s J0 | TestLucene42DocValuesFormat.testTwoNumbersOneMissingWithMerging
   [junit4]    > Assumption #1: Codec does not support getDocsWithField
   [junit4] IGNOR/A 0.01s J0 | TestLucene42DocValuesFormat.testTwoBytesOneMissing
   [junit4]    > Assumption #1: Codec does not support getDocsWithField
   [junit4] IGNOR/A 0.01s J0 | TestLucene42DocValuesFormat.testTwoBytesOneMissingWithMerging
   [junit4]    > Assumption #1: Codec does not support getDocsWithField
   [junit4] IGNOR/A 0.01s J0 | TestLucene42DocValuesFormat.testLongMissingVsFieldCache
   [junit4]    > Assumption #1: Codec does not support getDocsWithField
   [junit4] IGNOR/A 0.01s J0 | TestLucene42DocValuesFormat.testThreeBytesOneMissingWithMerging
   [junit4]    > Assumption #1: Codec does not support getDocsWithField
   [junit4] IGNOR/A 0.01s J0 | TestLucene42DocValuesFormat.testThreeNumbersOneMissingWithMerging
   [junit4]    > Assumption #1: Codec does not support getDocsWithField
   [junit4] IGNOR/A 0.01s J0 | TestLucene42DocValuesFormat.testShortMissingVsFieldCache
   [junit4]    > Assumption #1: Codec does not support getDocsWithField
   [junit4] IGNOR/A 0.01s J0 | TestLucene42DocValuesFormat.testTwoNumbersOneMissing
   [junit4]    > Assumption #1: Codec does not support getDocsWithField
   [junit4] Completed on J0 in 22.68s, 81 tests, 11 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestPrefixRandom
   [junit4] Completed on J0 in 0.63s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.store.TestLock
   [junit4] Completed on J0 in 0.02s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterConfig
   [junit4] Completed on J0 in 0.05s, 9 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.automaton.TestMinimize
   [junit4] Completed on J1 in 4.50s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestDocumentWriter
   [junit4] Completed on J1 in 0.24s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestBooleanQuery
   [junit4] Completed on J1 in 0.89s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestBroadWord
   [junit4] Completed on J1 in 0.69s, 11 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestNRTThreads
   [junit4] Completed on J0 in 5.36s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestTransactionRollback
   [junit4] Completed on J0 in 0.50s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestRamUsageEstimator
   [junit4] Completed on J0 in 0.02s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.document.TestDateTools
   [junit4] Completed on J0 in 0.06s, 5 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestPostingsFormat
   [junit4] Completed on J0 in 2.14s, 11 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.analysis.TestMockCharFilter
   [junit4] Completed on J0 in 0.01s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestDisjunctionMaxQuery
   [junit4] Completed on J0 in 0.47s, 11 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.automaton.TestDeterminizeLexicon
   [junit4] Completed on J0 in 2.92s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestCloseableThreadLocal
   [junit4] Completed on J0 in 0.02s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.automaton.TestSpecialOperations
   [junit4] Completed on J0 in 0.21s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestSubScorerFreqs
   [junit4] Completed on J0 in 0.05s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestBagOfPositions
   [junit4] Completed on J0 in 2.80s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestNoMergeScheduler
   [junit4] Completed on J0 in 0.02s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexableField
   [junit4] Completed on J0 in 0.59s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestSentinelIntSet
   [junit4] Completed on J0 in 0.18s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestScorerPerf
   [junit4] Completed on J0 in 17.00s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestFilteredQuery
   [junit4] Completed on J0 in 0.35s, 12 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.junitcompat.TestSystemPropertiesInvariantRule
   [junit4] Completed on J0 in 0.08s, 5 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestRegexpQuery
   [junit4] Completed on J0 in 0.06s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestRollback
   [junit4] Completed on J0 in 0.02s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.codecs.lucene40.TestLucene40PostingsReader
   [junit4] Completed on J0 in 3.44s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestReaderClosed
   [junit4] Completed on J0 in 0.03s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.payloads.TestPayloadTermQuery
   [junit4] Completed on J0 in 0.50s, 6 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestByteSlices
   [junit4] Completed on J0 in 5.55s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestStressNRT
   [junit4] Completed on J0 in 2.78s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestTwoPhaseCommitTool
   [junit4] Completed on J0 in 0.02s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestTimSorter
   [junit4] Completed on J0 in 0.18s, 9 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestCompoundFile
   [junit4] Completed on J0 in 0.98s, 18 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestMaxTermFrequency
   [junit4] Completed on J0 in 0.14s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterOnDiskFull
   [junit4] Completed on J0 in 0.84s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.junitcompat.TestFailOnFieldCacheInsanity
   [junit4] Completed on J0 in 0.03s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.codecs.compressing.TestFastCompressionMode
   [junit4] Completed on J0 in 1.26s, 11 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.analysis.tokenattributes.TestCharTermAttributeImpl
   [junit4] Completed on J0 in 0.42s, 12 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.TestMergeSchedulerExternal
   [junit4] Completed on J0 in 0.40s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestBytesRef
   [junit4] Completed on J0 in 0.03s, 5 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.spans.TestSpanExplanationsOfNonMatches
   [junit4] Completed on J0 in 0.12s, 31 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestParallelAtomicReader
   [junit4] Completed on J0 in 0.13s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestQueryBuilder
   [junit4] Completed on J0 in 0.06s, 19 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestWildcardRandom
   [junit4] Completed on J0 in 0.11s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.automaton.TestDeterminism
   [junit4] Completed on J0 in 0.60s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.Test2BTerms
   [junit4] Completed on J0 in 0.00s, 1 test, 1 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestAllFilesHaveCodecHeader
   [junit4] Completed on J0 in 0.18s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.TestExternalCodecs
   [junit4] Completed on J0 in 0.94s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.TestSearchForDuplicates
   [junit4] Completed on J0 in 0.17s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.junitcompat.TestExceptionInBeforeClassHooks
   [junit4] Completed on J0 in 0.15s, 3 tests
   [junit4] 
   [junit4] HEARTBEAT J1 PID(27937@serv1.sd-datasolutions.de): 2014-03-19T19:17:08, stalled for 66.8s at: TestMixedDocValuesUpdates.testTonsOfUpdates
   [junit4] Suite: org.apache.lucene.search.TestSearchAfter
   [junit4] Completed on J0 in 26.67s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestDocBoost
   [junit4] Completed on J0 in 0.02s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestStressAdvance
   [junit4] Completed on J0 in 0.74s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestConstantScoreQuery
   [junit4] Completed on J0 in 0.04s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestBooleanMinShouldMatch
   [junit4] Completed on J0 in 1.96s, 17 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestFilterIterator
   [junit4] Completed on J0 in 0.03s, 8 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.codecs.lucene40.TestLucene40StoredFieldsFormat
   [junit4] IGNOR/A 0.01s J0 | TestLucene40StoredFieldsFormat.testBigDocuments
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly())
   [junit4] Completed on J0 in 2.85s, 10 tests, 1 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.automaton.TestCompiledAutomaton
   [junit4] Completed on J0 in 0.28s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterForceMerge
   [junit4] Completed on J0 in 1.53s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterReader
   [junit4] Completed on J0 in 6.69s, 22 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestSegmentTermEnum
   [junit4] Completed on J0 in 0.08s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestNRTReaderWithThreads
   [junit4] Completed on J0 in 1.20s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterCommit
   [junit4] Completed on J0 in 1.63s, 12 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.store.TestLockFactory
   [junit4] IGNOR/A 0.00s J0 | TestLockFactory.testStressLocksNativeFSLockFactory
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly())
   [junit4] IGNOR/A 0.00s J0 | TestLockFactory.testStressLocks
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly())
   [junit4] Completed on J0 in 1.06s, 11 tests, 2 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestMixedDocValuesUpdates
   [junit4] Completed on J1 in 132.33s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.store.TestByteArrayDataInput
   [junit4] Completed on J1 in 0.06s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestOpenBitSet
   [junit4] Completed on J1 in 23.15s, 9 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestPrefixCodedTerms
   [junit4] Completed on J1 in 0.30s, 5 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestTotalHitCountCollector
   [junit4] Completed on J1 in 0.02s, 1 test
   [junit4] 
   [junit4] HEARTBEAT J0 PID(27938@serv1.sd-datasolutions.de): 2014-03-19T19:18:44, stalled for 69.3s at: TestNumericDocValuesUpdates.testTonsOfUpdates
   [junit4] Suite: org.apache.lucene.store.TestMultiMMap
   [junit4] Completed on J1 in 8.38s, 11 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestCachingWrapperFilter
   [junit4] Completed on J1 in 0.40s, 11 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestVersionComparator
   [junit4] Completed on J1 in 0.02s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.codecs.compressing.TestHighCompressionMode
   [junit4] Completed on J1 in 4.10s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.store.TestCopyBytes
   [junit4] Completed on J1 in 0.82s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestSnapshotDeletionPolicy
   [junit4] Completed on J1 in 1.37s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestDocValuesScoring
   [junit4] Completed on J1 in 0.07s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestAtomicUpdate
   [junit4] Completed on J1 in 5.11s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestIntroSorter
   [junit4] Completed on J1 in 0.20s, 9 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.packed.TestEliasFanoSequence
   [junit4] Completed on J1 in 2.90s, 24 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterWithThreads
   [junit4] Completed on J1 in 4.87s, 12 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestSimpleExplanationsOfNonMatches
   [junit4] Completed on J1 in 0.24s, 69 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestFieldCache
   [junit4] Completed on J1 in 2.20s, 10 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestCharsRef
   [junit4] Completed on J1 in 0.07s, 8 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.codecs.perfield.TestPerFieldPostingsFormat
   [junit4] Completed on J1 in 1.76s, 11 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestByteBlockPool
   [junit4] Completed on J1 in 0.17s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestNumericDocValuesUpdates
   [junit4] Completed on J0 in 113.75s, 31 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestUniqueTermCount
   [junit4] Completed on J0 in 0.10s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterExceptions
   [junit4] Completed on J1 in 29.64s, 27 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestTermVectors
   [junit4] Completed on J1 in 0.37s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestTermdocPerf
   [junit4] Completed on J1 in 0.02s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestThreadedForceMerge
   [junit4] Completed on J1 in 0.51s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestIOUtils
   [junit4] Completed on J1 in 0.01s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexFileDeleter
   [junit4] Completed on J1 in 0.16s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestSegmentMerger
   [junit4] Completed on J1 in 0.99s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestCodecs
   [junit4] Completed on J1 in 1.00s, 5 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestSameTokenSamePosition
   [junit4] Completed on J1 in 0.09s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.Test4GBStoredFields
   [junit4] IGNOR/A 0.02s J1 | Test4GBStoredFields.test
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly())
   [junit4] Completed on J1 in 0.02s, 1 test, 1 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestDocCount
   [junit4] Completed on J1 in 0.10s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestParallelCompositeReader
   [junit4] Completed on J1 in 0.70s, 13 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestRollingBuffer
   [junit4] Completed on J1 in 0.52s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterDelete
   [junit4] IGNOR/A 0.00s J1 | TestIndexWriterDelete.testApplyDeletesOnFlush
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly())
   [junit4] Completed on J1 in 6.79s, 22 tests, 1 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestSumDocFreq
   [junit4] Completed on J1 in 0.20s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestFuzzyQuery
   [junit4] Completed on J1 in 0.17s, 6 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.payloads.TestPayloadNearQuery
   [junit4] Completed on J1 in 0.29s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestTerm
   [junit4] Completed on J1 in 0.01s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestDocTermOrdsRangeFilter
   [junit4] Completed on J1 in 1.31s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestSmallFloat
   [junit4] Completed on J1 in 0.13s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexWriter
   [junit4] Completed on J0 in 32.94s, 57 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.analysis.TestGraphTokenizers
   [junit4] Completed on J1 in 17.59s, 21 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestTryDelete
   [junit4] Completed on J1 in 0.02s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.store.TestNRTCachingDirectory
   [junit4] Completed on J1 in 1.95s, 5 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.spans.TestSpansAdvanced
   [junit4] Completed on J1 in 0.08s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestPrefixQuery
   [junit4] Completed on J1 in 0.02s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestBooleanOr
   [junit4] Completed on J1 in 1.22s, 6 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestMatchAllDocsQuery
   [junit4] Completed on J1 in 0.03s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestNGramPhraseQuery
   [junit4] Completed on J1 in 0.02s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestMultiTermConstantScore
   [junit4] Completed on J1 in 1.14s, 8 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestDirectoryReaderReopen
   [junit4] Completed on J1 in 1.49s, 6 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.automaton.TestBasicOperations
   [junit4] Completed on J1 in 0.62s, 8 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestLogMergePolicy
   [junit4] Completed on J1 in 0.04s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.analysis.TestCharFilter
   [junit4] Completed on J1 in 0.03s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.spans.TestNearSpansOrdered
   [junit4] Completed on J1 in 0.18s, 10 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.spans.TestSpanSearchEquivalence
   [junit4] Completed on J1 in 0.56s, 8 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.store.TestDirectory
   [junit4] IGNOR/A 0.00s J1 | TestDirectory.testThreadSafety
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly())
   [junit4] Completed on J1 in 0.74s, 8 tests, 1 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.codecs.perfield.TestPerFieldDocValuesFormat
   [junit4] IGNOR/A 0.01s J0 | TestPerFieldDocValuesFormat.testHugeBinaryValueLimit
   [junit4]    > Assumption #1: test requires codec with limits on max binary field length
   [junit4] Completed on J0 in 28.40s, 79 tests, 1 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestTermVectorsFormat
   [junit4] Completed on J0 in 7.84s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestDocValuesFormat
   [junit4] IGNOR/A 0.01s J1 | TestDocValuesFormat.testHugeBinaryValueLimit
   [junit4]    > Assumption #1: test requires codec with limits on max binary field length
   [junit4] Completed on J1 in 19.78s, 78 tests, 1 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.analysis.TestCachingTokenFilter
   [junit4] Completed on J1 in 0.07s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestSortDocValues
   [junit4] Completed on J1 in 0.35s, 23 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestRecyclingIntBlockAllocator
   [junit4] Completed on J1 in 0.10s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterMerging
   [junit4] Completed on J1 in 11.20s, 6 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.fst.TestFSTs
   [junit4] IGNOR/A 0.00s J1 | TestFSTs.testBigSet
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly())
   [junit4] Completed on J1 in 26.25s, 19 tests, 1 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.codecs.compressing.TestCompressingTermVectorsFormat
   [junit4] Completed on J0 in 39.47s, 40 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.fst.TestBytesStore
   [junit4] Completed on J0 in 9.15s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestNumericRangeQuery64
   [junit4] Completed on J1 in 15.26s, 34 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestNumericRangeQuery32
   [junit4] Completed on J1 in 4.05s, 27 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestBackwardsCompatibility
   [junit4] Completed on J1 in 12.65s, 15 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.packed.TestPackedInts
   [junit4] IGNOR/A 0.00s J0 | TestPackedInts.testBlockReaderOverflow
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly())
   [junit4] Completed on J0 in 25.67s, 27 tests, 3 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestAddIndexes
   [junit4] Completed on J1 in 4.08s, 23 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.codecs.lucene41.TestBlockPostingsFormat3
   [junit4] Completed on J1 in 7.73s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestDeletionPolicy
   [junit4] Completed on J1 in 4.11s, 6 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestShardSearching
   [junit4] Completed on J0 in 13.66s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterMergePolicy
   [junit4] Completed on J0 in 2.42s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.codecs.lucene40.TestLucene40DocValuesFormat
   [junit4] IGNOR/A 0.01s J1 | TestLucene40DocValuesFormat.testSortedSetOneValue
   [junit4]    > Assumption #1: Codec does not support SORTED_SET
   [junit4] IGNOR/A 0.01s J1 | TestLucene40DocValuesFormat.testByteMissingVsFieldCache
   [junit4]    > Assumption #1: Codec does not support getDocsWithField
   [junit4] IGNOR/A 0.01s J1 | TestLucene40DocValuesFormat.testThreeBytesOneMissingWithMerging
   [junit4]    > Assumption #1: Codec does not support getDocsWithField
   [junit4] IGNOR/A 0.01s J1 | TestLucene40DocValuesFormat.testSortedSetTwoFields
   [junit4]    > Assumption #1: Codec does not support SORTED_SET
   [junit4] IGNOR/A 0.01s J1 | TestLucene40DocValuesFormat.testSortedSetMergeAwayAllValues
   [junit4]    > Assumption #1: Codec does not support SORTED_SET
   [junit4] IGNOR/A 0.01s J1 | TestLucene40DocValuesFormat.testTwoBytesOneMissingWithMerging
   [junit4]    > Assumption #1: Codec does not support getDocsWithField
   [junit4] IGNOR/A 0.01s J1 | TestLucene40DocValuesFormat.testTwoBytesOneMissing
   [junit4]    > Assumption #1: Codec does not support getDocsWithField
   [junit4] IGNOR/A 0.01s J1 | TestLucene40DocValuesFormat.testSortedSetTwoDocumentsLastMissing
   [junit4]    > Assumption #1: Codec does not support SORTED_SET
   [junit4] IGNOR/A 0.01s J1 | TestLucene40DocValuesFormat.testThreads2
   [junit4]    > Assumption #1: Codec does not support getDocsWithField
   [junit4] IGNOR/A 0.01s J1 | TestLucene40DocValuesFormat.testSortedSetTwoDocumentsMerged
   [junit4]    > Assumption #1: Codec does not support SORTED_SET
   [junit4] IGNOR/A 0.01s J1 | TestLucene40DocValuesFormat.testIntMissingVsFieldCache
   [junit4]    > Assumption #1: Codec does not support getDocsWithField
   [junit4] IGNOR/A 0.01s J1 | TestLucene40DocValuesFormat.testSortedSetFixedLengthVsStoredFields
   [junit4]    > Assumption #1: Codec does not support SORTED_SET
   [junit4] IGNOR/A 0.01s J1 | TestLucene40DocValuesFormat.testTwoNumbersOneMissingWithMerging
   [junit4]    > Assumption #1: Codec does not support getDocsWithField
   [junit4] IGNOR/A 0.01s J1 | TestLucene40DocValuesFormat.testSortedSetThreeValuesTwoDocs
   [junit4]    > Assumption #1: Codec does not support SORTED_SET
   [junit4] IGNOR/A 0.00s J1 | TestLucene40DocValuesFormat.testSortedSetFixedLengthSingleValuedVsStoredFields
   [junit4]    > Assumption #1: Codec does not support SORTED_SET
   [junit4] IGNOR/A 0.01s J1 | TestLucene40DocValuesFormat.testSortedSetTwoValuesUnordered
   [junit4]    > Assumption #1: Codec does not support SORTED_SET
   [junit4] IGNOR/A 0.01s J1 | TestLucene40DocValuesFormat.testSortedSetVariableLengthSingleValuedVsStoredFields
   [junit4]    > Assumption #1: Codec does not support SORTED_SET
   [junit4] IGNOR/A 0.01s J1 | TestLucene40DocValuesFormat.testSortedSetFixedLengthVsUninvertedField
   [junit4]    > Assumption #1: Codec does not support SORTED_SET
   [junit4] IGNOR/A 0.01s J1 | TestLucene40DocValuesFormat.testSortedSetTwoDocumentsLastMissingMerge
   [junit4]    > Assumption #1: Codec does not support SORTED_SET
   [junit4] IGNOR/A 0.01s J1 | TestLucene40DocValuesFormat.testTwoNumbersOneMissing
   [junit4]    > Assumption #1: Codec does not support getDocsWithField
   [junit4] IGNOR/A 0.01s J1 | TestLucene40DocValuesFormat.testSortedSetTermsEnum
   [junit4]    > Assumption #1: Codec does not support SORTED_SET
   [junit4] IGNOR/A 0.01s J1 | TestLucene40DocValuesFormat.testLongMissingVsFieldCache
   [junit4]    > Assumption #1: Codec does not support getDocsWithField
   [junit4] IGNOR/A 0.00s J1 | TestLucene40DocValuesFormat.testShortMissingVsFieldCache
   [junit4]    > Assumption #1: Codec does not support getDocsWithField
   [junit4] IGNOR/A 0.00s J1 | TestLucene40DocValuesFormat.testSortedSetTwoDocumentsFirstMissingMerge
   [junit4]    > Assumption #1: Codec does not support SORTED_SET
   [junit4] IGNOR/A 0.01s J1 | TestLucene40DocValuesFormat.testSortedSetTwoDocumentsFirstMissing
   [junit4]    > Assumption #1: Codec does not support SORTED_SET
   [junit4] IGNOR/A 0.01s J1 | TestLucene40DocValuesFormat.testSortedSetVariableLengthVsStoredFields
   [junit4]    > Assumption #1: Codec does not support SORTED_SET
   [junit4] IGNOR/A 0.01s J1 | TestLucene40DocValuesFormat.testSortedSetVariableLengthVsUninvertedField
   [junit4]    > Assumption #1: Codec does not support SORTED_SET
   [junit4] IGNOR/A 0.01s J1 | TestLucene40DocValuesFormat.testThreeNumbersOneMissingWithMerging
   [junit4]    > Assumption #1: Codec does not support getDocsWithField
   [junit4] IGNOR/A 0.01s J1 | TestLucene40DocValuesFormat.testSortedSetTwoValues
   [junit4]    > Assumption #1: Codec does not support SORTED_SET
   [junit4] Completed on J1 in 9.08s, 78 tests, 29 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestPersistentSnapshotDeletionPolicy
   [junit4] Completed on J1 in 1.61s, 13 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestNorms
   [junit4] Completed on J1 in 0.85s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestTimeLimitingCollector
   [junit4] Completed on J1 in 2.68s, 6 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestDuelingCodecs
   [junit4] Completed on J0 in 14.23s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestNeverDelete
   [junit4] Completed on J0 in 4.91s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestTopDocsMerge
   [junit4] Completed on J0 in 29.05s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestLongPostings
   [junit4] Completed on J0 in 3.03s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestDocumentsWriterStallControl
   [junit4] Completed on J1 in 41.45s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestRollingUpdates
   [junit4] Completed on J1 in 1.91s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestStressIndexing
   [junit4] Completed on J0 in 3.39s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.codecs.lucene41.TestBlockPostingsFormat2
   [junit4] Completed on J1 in 1.22s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterUnicode
   [junit4] Completed on J0 in 4.05s, 5 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestArrayUtil
   [junit4] Completed on J1 in 3.65s, 11 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.automaton.TestLevenshteinAutomata
   [junit4] Completed on J0 in 0.56s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestConsistentFieldNumbers
   [junit4] Completed on J0 in 2.86s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestTermsEnum
   [junit4] Completed on J1 in 9.37s, 9 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestWeakIdentityMap
   [junit4] Completed on J0 in 6.40s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestSimpleExplanations
   [junit4] Completed on J1 in 1.09s, 69 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestPhraseQuery
   [junit4] Completed on J0 in 1.38s, 18 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestOmitNorms
   [junit4] Completed on J0 in 0.38s, 5 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestCollectionUtil
   [junit4] Completed on J1 in 9.47s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestComplexExplanations
   [junit4] Completed on J1 in 0.73s, 22 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestFixedBitSet
   [junit4] Completed on J0 in 11.62s, 11 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestRegexpRandom2
   [junit4] Completed on J0 in 3.18s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestSegmentReader
   [junit4] Completed on J0 in 0.63s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.codecs.compressing.TestFastDecompressionMode
   [junit4] Completed on J0 in 2.33s, 11 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestFieldCacheRewriteMethod
   [junit4] Completed on J0 in 1.81s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.analysis.TestMockAnalyzer
   [junit4] Completed on J1 in 18.59s, 17 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestCrash
   [junit4] Completed on J1 in 0.85s, 5 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestSegmentTermDocs
   [junit4] Completed on J1 in 0.41s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestPagedBytes
   [junit4] Completed on J0 in 10.04s, 3 tests, 1 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.codecs.perfield.TestPerFieldPostingsFormat2
   [junit4] Completed on J1 in 0.89s, 5 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestSort
   [junit4] Completed on J1 in 0.52s, 43 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.codecs.lucene41.TestBlockPostingsFormat
   [junit4] Completed on J0 in 1.64s, 11 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.codecs.lucene40.TestBitVector
   [junit4] Completed on J1 in 0.91s, 9 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestFieldCacheSanityChecker
   [junit4] Completed on J0 in 0.43s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.analysis.TestToken
   [junit4] Completed on J1 in 0.70s, 10 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestSimpleSearchEquivalence
   [junit4] Completed on J0 in 0.34s, 12 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestDocValuesIndexing
   [junit4] IGNOR/A 0.01s J0 | TestDocValuesIndexing.testTooLargeTermSortedSetBytes
   [junit4]    > Assumption #1: codec does not support SORTED_SET
   [junit4] Completed on J0 in 0.66s, 30 tests, 1 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestCustomSearcherSort
   [junit4] Completed on J1 in 1.30s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.similarities.TestSimilarityBase
   [junit4] Completed on J0 in 0.64s, 30 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.store.TestHugeRamFile
   [junit4] Completed on J1 in 0.71s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestTermsEnum2
   [junit4] Completed on J0 in 1.49s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.Test2BDocs
   [junit4] Completed on J0 in 0.54s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.spans.TestSpans
   [junit4] Completed on J0 in 0.51s, 26 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestNumericUtils
   [junit4] Completed on J1 in 2.50s, 12 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestPostingsOffsets
   [junit4] Completed on J1 in 1.45s, 11 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestCustomNorms
   [junit4] Completed on J0 in 2.32s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestMixedCodecs
   [junit4] Completed on J1 in 1.87s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.codecs.lucene40.TestLucene40PostingsFormat
   [junit4] Completed on J0 in 2.16s, 11 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.spans.TestSpanExplanations
   [junit4] Completed on J0 in 0.58s, 31 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestMaxFailuresRule
   [junit4] Completed on J0 in 1.01s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.automaton.TestUTF32ToUTF8
   [junit4] Completed on J1 in 4.33s, 5 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestStressIndexing2
   [junit4] Completed on J0 in 1.59s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.store.TestRAMDirectory
   [junit4] Completed on J1 in 0.28s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestDocsAndPositions
   [junit4] Completed on J0 in 0.64s, 6 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestRamUsageEstimatorOnWildAnimals
   [junit4] Completed on J1 in 0.64s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestOmitPositions
   [junit4] Completed on J1 in 0.15s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.payloads.TestPayloadExplanations
   [junit4] Completed on J1 in 0.20s, 5 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestFieldCacheRangeFilter
   [junit4] Completed on J0 in 0.67s, 8 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexInput
   [junit4] Completed on J0 in 0.59s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestBytesRefHash
   [junit4] Completed on J1 in 1.49s, 8 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestSloppyPhraseQuery2
   [junit4] Completed on J0 in 0.77s, 9 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestQueryWrapperFilter
   [junit4] Completed on J0 in 0.24s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestOmitTf
   [junit4] Completed on J1 in 0.50s, 6 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestTermRangeQuery
   [junit4] Completed on J1 in 0.06s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestDocumentsWriterDeleteQueue
   [junit4] Completed on J1 in 0.10s, 5 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestMultiFields
   [junit4] Completed on J0 in 2.98s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestForTooMuchCloning
   [junit4] Completed on J0 in 0.09s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestDoc
   [junit4] Completed on J0 in 0.11s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.similarities.TestSimilarity2
   [junit4] Completed on J0 in 0.12s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestTermRangeFilter
   [junit4] Completed on J0 in 1.09s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestLiveFieldValues
   [junit4] Completed on J1 in 4.55s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.spans.TestFieldMaskingSpanQuery
   [junit4] Completed on J0 in 0.29s, 11 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestNoDeletionPolicy
   [junit4] Completed on J1 in 0.11s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.spans.TestSpansAdvanced2
   [junit4] Completed on J0 in 0.07s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestPerSegmentDeletes
   [junit4] Completed on J1 in 0.09s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestPrefixInBooleanQuery
   [junit4] Completed on J1 in 0.37s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestMultiValuedNumericRangeQuery
   [junit4] Completed on J0 in 0.63s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestTopDocsCollector
   [junit4] Completed on J1 in 0.05s, 8 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestComplexExplanationsOfNonMatches
   [junit4] Completed on J0 in 0.09s, 22 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestUnicodeUtil
   [junit4] Completed on J1 in 0.47s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestSortRandom
   [junit4] Completed on J0 in 0.36s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestParallelReaderEmptyIndex
   [junit4] Completed on J0 in 0.02s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.codecs.lucene41.TestForUtil
   [junit4] Completed on J0 in 0.06s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.document.TestDocument
   [junit4] Completed on J0 in 0.05s, 12 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestAutomatonQuery
   [junit4] Completed on J1 in 0.91s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestTopFieldCollector
   [junit4] Completed on J0 in 0.06s, 8 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.store.TestFileSwitchDirectory
   [junit4] Completed on J1 in 0.04s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.store.TestWindowsMMap
   [junit4] Completed on J0 in 0.21s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestRegexpRandom
   [junit4] Completed on J1 in 0.05s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestMultiLevelSkipList
   [junit4] Completed on J0 in 0.06s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestPayloadsOnVectors
   [junit4] Completed on J1 in 0.08s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestFieldValueFilter
   [junit4] Completed on J0 in 0.07s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.spans.TestSpanMultiTermQueryWrapper
   [junit4] Completed on J1 in 0.12s, 8 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestSizeBoundedForceMerge
   [junit4] Completed on J0 in 0.06s, 11 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.junitcompat.TestBeforeAfterOverrides
   [junit4] Completed on J1 in 0.04s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestElevationComparator
   [junit4] Completed on J0 in 0.02s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestFilteredSearch
   [junit4] Completed on J1 in 0.12s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestBinaryTerms
   [junit4] Completed on J0 in 0.07s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestConjunctions
   [junit4] Completed on J1 in 0.02s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestMultiTermQueryRewrites
   [junit4] Completed on J0 in 0.03s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestPositionIncrement
   [junit4] Completed on J1 in 0.02s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.junitcompat.TestSameRandomnessLocalePassedOrNot
   [junit4] Completed on J0 in 0.02s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.document.TestField
   [junit4] Completed on J1 in 0.06s, 18 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestDateSort
   [junit4] Completed on J0 in 0.01s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestSimilarity
   [junit4] Completed on J1 in 0.01s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestVirtualMethod
   [junit4] Completed on J0 in 0.01s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestSimilarityProvider
   [junit4] Completed on J1 in 0.04s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.analysis.TestNumericTokenStream
   [junit4] Completed on J0 in 0.02s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestSetOnce
   [junit4] Completed on J1 in 0.03s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestFilterAtomicReader
   [junit4] Completed on J0 in 0.02s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.TestSearch
   [junit4] Completed on J1 in 0.08s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.junitcompat.TestSeedFromUncaught
   [junit4] Completed on J0 in 0.03s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestPriorityQueue
   [junit4] Completed on J1 in 0.05s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.Test2BPostings
   [junit4] IGNOR/A 0.00s J0 | Test2BPostings.test
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly())
   [junit4] Completed on J0 in 0.00s, 1 test, 1 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.document.TestBinaryDocument
   [junit4] Completed on J1 in 0.02s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestAutomatonQueryUnicode
   [junit4] Completed on J0 in 0.01s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestFieldCacheTermsFilter
   [junit4] Completed on J1 in 0.02s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestAttributeSource
   [junit4] Completed on J0 in 0.02s, 5 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestRecyclingByteBlockAllocator
   [junit4] Completed on J1 in 0.04s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.store.TestMockDirectoryWrapper
   [junit4] Completed on J0 in 0.02s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.junitcompat.TestFailIfUnreferencedFiles
   [junit4] Completed on J1 in 0.02s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.spans.TestSpanFirstQuery
   [junit4] Completed on J0 in 0.01s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.junitcompat.TestJUnitRuleOrder
   [junit4] Completed on J1 in 0.02s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIsCurrent
   [junit4] Completed on J0 in 0.02s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestNoMergePolicy
   [junit4] Completed on J1 in 0.02s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterLockRelease
   [junit4] Completed on J0 in 0.01s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.junitcompat.TestLeaveFilesIfTestFails
   [junit4] Completed on J1 in 0.02s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.TestDemo
   [junit4] Completed on J0 in 0.02s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.TestAssertions
   [junit4] Completed on J1 in 0.01s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestFieldInfos
   [junit4] Completed on J0 in 0.02s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestNot
   [junit4] Completed on J1 in 0.01s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestExplanations
   [junit4] Completed on J0 in 0.06s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterOnJRECrash
   [junit4] IGNOR/A 0.00s J1 | TestIndexWriterOnJRECrash.testNRTThreads
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly())
   [junit4] Completed on J1 in 0.00s, 1 test, 1 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestPrefixFilter
   [junit4] Completed on J0 in 0.06s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestVersion
   [junit4] Completed on J1 in 0.02s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.junitcompat.TestCodecReported
   [junit4] Completed on J0 in 0.01s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.MultiCollectorTest
   [junit4] Completed on J1 in 0.02s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestScoreCachingWrappingScorer
   [junit4] Completed on J0 in 0.01s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestIntsRef
   [junit4] Completed on J1 in 0.02s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.TestWorstCaseTestBehavior
   [junit4] Completed on J0 in 0.01s, 6 tests, 6 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.analysis.tokenattributes.TestSimpleAttributeImpl
   [junit4] Completed on J1 in 0.02s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.Test2BPositions
   [junit4] Completed on J0 in 0.00s, 1 test, 1 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.Test2BPostingsBytes
   [junit4] Completed on J1 in 0.00s, 1 test, 1 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestPositiveScoresOnlyCollector
   [junit4] Completed on J0 in 0.01s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestTopScoreDocCollector
   [junit4] Completed on J1 in 0.01s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestNewestSegment
   [junit4] Completed on J0 in 0.01s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.analysis.TestReusableStringReader
   [junit4] Completed on J0 in 0.01s, 1 test
   [junit4] 
   [junit4]

[...truncated too long message...]

CloudSolrServerTest-1395260108083/jetty3/index;done=false>>]
   [junit4]   2> 126630 T192 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:259)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 126630 T173 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395260108083/jetty3/index
   [junit4]   2> 126630 T192 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 126630 T173 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395260108083/jetty3 [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395260108083/jetty3;done=false>>]
   [junit4]   2> 126631 T173 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395260108083/jetty3
   [junit4]   2> 126631 T173 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 126631 T173 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 126632 T173 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 126632 T173 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:259)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> NOTE: test params are: codec=Lucene46: {_version_=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), range_facet_si=PostingsFormat(name=Asserting), multiDefault=PostingsFormat(name=Asserting), intDefault=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), id=PostingsFormat(name=FSTPulsing41), timestamp=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), range_facet_l=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), a_t=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), text=PostingsFormat(name=FSTOrdPulsing41), range_facet_sl=PostingsFormat(name=FSTOrdPulsing41)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=sl, timezone=America/Cuiaba
   [junit4]   2> NOTE: Linux 3.8.0-37-generic amd64/Oracle Corporation 1.7.0_51 (64-bit)/cpus=8,threads=1,free=64382744,total=126550016
   [junit4]   2> NOTE: All tests run in this JVM: [SolrPingTest, AnlysisResponseBaseTest, TestJavaBinCodec, LargeVolumeBinaryJettyTest, TestXMLEscaping, LargeVolumeEmbeddedTest, SolrExampleXMLTest, TestRetryUtil, ClientUtilsTest, TestDocumentObjectBinder, SolrExceptionTest, DocumentAnalysisResponseTest, CloudSolrServerTest]
   [junit4] Completed on J0 in 101.23s, 2 tests, 1 error <<< FAILURES!
   [junit4] 
   [junit4] Suite: org.apache.solr.client.solrj.embedded.SolrExampleStreamingTest
   [junit4] Completed on J1 in 19.05s, 27 tests
   [junit4] 
   [junit4] Suite: org.apache.solr.client.solrj.embedded.SolrExampleStreamingBinaryTest
   [junit4] Completed on J0 in 19.72s, 27 tests
   [junit4] 
   [junit4] Suite: org.apache.solr.client.solrj.embedded.SolrExampleEmbeddedTest
   [junit4] Completed on J1 in 6.81s, 26 tests
   [junit4] 
   [junit4] Suite: org.apache.solr.client.solrj.impl.BasicHttpSolrServerTest
   [junit4] Completed on J0 in 3.11s, 10 tests
   [junit4] 
   [junit4] Suite: org.apache.solr.client.solrj.TestBatchUpdate
   [junit4] Completed on J1 in 2.12s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.solr.client.solrj.embedded.JettyWebappTest
   [junit4] Completed on J0 in 1.56s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.solr.client.solrj.embedded.MultiCoreExampleJettyTest
   [junit4] Completed on J1 in 2.56s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.solr.client.solrj.response.TestSpellCheckResponse
   [junit4] Completed on J0 in 2.14s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.solr.common.params.SolrParamTest
   [junit4] Completed on J0 in 0.02s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.solr.client.solrj.embedded.LargeVolumeJettyTest
   [junit4] Completed on J1 in 1.25s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.solr.client.solrj.embedded.MultiCoreEmbeddedTest
   [junit4] Completed on J0 in 0.79s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.solr.client.solrj.embedded.TestEmbeddedSolrServer
   [junit4] Completed on J0 in 0.59s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.solr.client.solrj.response.TermsResponseTest
   [junit4] Completed on J1 in 1.73s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.solr.client.solrj.request.TestCoreAdmin
   [junit4] Completed on J0 in 0.79s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.solr.common.util.ContentStreamTest
   [junit4] Completed on J1 in 0.09s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.solr.common.params.ModifiableSolrParamsTest
   [junit4] Completed on J0 in 0.02s, 5 tests
   [junit4] 
   [junit4] Suite: org.apache.solr.client.solrj.impl.HttpClientUtilTest
   [junit4] Completed on J1 in 0.01s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.solr.common.util.IteratorChainTest
   [junit4] Completed on J0 in 0.03s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.solr.client.solrj.request.TestUpdateRequestCodec
   [junit4] Completed on J1 in 0.02s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.solr.client.solrj.response.FieldAnalysisResponseTest
   [junit4] Completed on J0 in 0.02s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.solr.common.util.NamedListTest
   [junit4] Completed on J1 in 0.03s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.solr.client.solrj.response.NoOpResponseParserTest
   [junit4] Completed on J0 in 1.03s, 3 tests
   [junit4] 
   [junit4] 
   [junit4] Tests with failures:
   [junit4]   - org.apache.solr.client.solrj.impl.CloudSolrServerTest.testShutdown
   [junit4] 
   [junit4] 
   [junit4] JVM J0:     1.38 ..   162.24 =   160.87s
   [junit4] JVM J1:     1.38 ..   161.06 =   159.69s
   [junit4] Execution time total: 2 minutes 42 seconds
   [junit4] Tests summary: 49 suites, 283 tests, 1 error

BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:202: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1275: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 49 suites, 283 tests, 1 error

Total time: 62 minutes 23 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_51 -XX:-UseCompressedOops -XX:+UseSerialGC -XX:-UseSuperWord
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_60-ea-b10) - Build # 9848 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9848/
Java: 32bit/jdk1.7.0_60-ea-b10 -client -XX:+UseConcMarkSweepGC

2 tests failed.
REGRESSION:  org.apache.solr.handler.component.DistributedQueryComponentCustomSortTest.testDistribSearch

Error Message:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://[ff01::114]:33332/ugek/em, https://127.0.0.1:49411/ugek/em, https://[ff01::083]:33332/ugek/em, https://[ff01::213]:33332/ugek/em]

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://[ff01::114]:33332/ugek/em, https://127.0.0.1:49411/ugek/em, https://[ff01::083]:33332/ugek/em, https://[ff01::213]:33332/ugek/em]
	at __randomizedtesting.SeedInfo.seed([DB25039498A55DBD:5AC38D8CEFFA3D81]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:504)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:203)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.BaseDistributedSearchTestCase.queryServer(BaseDistributedSearchTestCase.java:515)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:563)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:545)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:524)
	at org.apache.solr.handler.component.DistributedQueryComponentCustomSortTest.doTest(DistributedQueryComponentCustomSortTest.java:70)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:826)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:862)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)


REGRESSION:  org.apache.solr.handler.component.DistributedSuggestComponentTest.testDistribSearch

Error Message:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:52715, https://[ff01::114]:33332, https://[ff01::083]:33332, https://[ff01::213]:33332]

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:52715, https://[ff01::114]:33332, https://[ff01::083]:33332, https://[ff01::213]:33332]
	at __randomizedtesting.SeedInfo.seed([DB25039498A55DBD:5AC38D8CEFFA3D81]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:504)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:203)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.BaseDistributedSearchTestCase.queryServer(BaseDistributedSearchTestCase.java:515)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:563)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:545)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:524)
	at org.apache.solr.handler.component.DistributedSuggestComponentTest.doTest(DistributedSuggestComponentTest.java:121)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:876)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:826)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:862)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 10036 lines...]
   [junit4] Suite: org.apache.solr.handler.component.DistributedSuggestComponentTest
   [junit4]   2> 147198 T215 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 147198 T215 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 147206 T215 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 147214 T215 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 147338 T215 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 147342 T215 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:45637
   [junit4]   2> 147344 T215 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 147345 T215 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 147345 T215 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr
   [junit4]   2> 147345 T215 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/'
   [junit4]   2> 147375 T215 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/solr.xml
   [junit4]   2> 147448 T215 oasc.CoreContainer.<init> New CoreContainer 3360953
   [junit4]   2> 147449 T215 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/]
   [junit4]   2> 147450 T215 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 147451 T215 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: https
   [junit4]   2> 147451 T215 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 147452 T215 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 147452 T215 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 147452 T215 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 147457 T215 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 147458 T215 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 147458 T215 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 147462 T215 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 147462 T215 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 147463 T215 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 147469 T226 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1
   [junit4]   2> 147469 T226 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 147470 T226 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 147470 T226 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 147470 T226 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 147530 T226 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 147613 T226 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-suggestercomponent.xml
   [junit4]   2> 147614 T226 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 147622 T226 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 148212 T226 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 148226 T226 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 148228 T226 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 148237 T226 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 148241 T226 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 148243 T226 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 148248 T226 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 148249 T226 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 148249 T226 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 148249 T226 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 148250 T226 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 148250 T226 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
   [junit4]   2> 148257 T226 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/, dataDir=./org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/control/data/
   [junit4]   2> 148257 T226 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 148257 T226 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/control/data
   [junit4]   2> 148258 T226 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/control/data/index/
   [junit4]   2> 148258 T226 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 148259 T226 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/control/data/index
   [junit4]   2> 148260 T226 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=32, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=28.2919921875, floorSegmentMB=0.685546875, forceMergeDeletesPctAllowed=4.651501728419072, segmentsPerTier=44.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 148262 T226 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=NRTCachingDirectory(NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/control/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@25ec7; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
   [junit4]   2> 148262 T226 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 148283 T226 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 148284 T226 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 148284 T226 oasc.RequestHandlers.initHandlersFromConfig created /suggest: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 148284 T226 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 148287 T226 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 148295 T226 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 148296 T226 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 148296 T226 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=50, maxMergeAtOnceExplicit=13, maxMergedSegmentMB=54.0068359375, floorSegmentMB=1.7255859375, forceMergeDeletesPctAllowed=9.61680111966303, segmentsPerTier=14.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.33166774602003857
   [junit4]   2> 148297 T226 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=NRTCachingDirectory(NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/control/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@25ec7; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
   [junit4]   2> 148298 T226 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 148298 T226 oass.SolrIndexSearcher.<init> Opening Searcher@18ef251[collection1] main
   [junit4]   2> 148298 T226 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf
   [junit4]   2> 148298 T226 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/}
   [junit4]   2> 148299 T226 oasr.ManagedResourceStorage.load Reading _rest_managed.json using file:dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf
   [junit4]   2> 148299 T226 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
   [junit4]   2> 148299 T226 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 148299 T226 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 148299 T226 oashc.SuggestComponent.inform Initializing SuggestComponent
   [junit4]   2> 148302 T226 oasss.SolrSuggester.init init: {name=suggest_fuzzy_with_high_freq_dict,lookupImpl=FuzzyLookupFactory,field=cat,storeDir=suggest_fuzzy_with_high_freq_dict,suggestAnalyzerFieldType=text,buildOnCommit=true,threshold=0.0}
   [junit4]   2> 148331 T226 oasss.SolrSuggester.init No dictionaryImpl parameter was provided falling back to org.apache.solr.spelling.suggest.HighFrequencyDictionaryFactory
   [junit4]   2> 148331 T226 oasss.SolrSuggester.init Dictionary loaded with params: {name=suggest_fuzzy_with_high_freq_dict,lookupImpl=FuzzyLookupFactory,field=cat,storeDir=suggest_fuzzy_with_high_freq_dict,suggestAnalyzerFieldType=text,buildOnCommit=true,threshold=0.0}
   [junit4]   2> 148332 T226 oashc.SuggestComponent.inform Registering newSearcher listener for suggester: suggest_fuzzy_with_high_freq_dict
   [junit4]   2> 148333 T226 oasss.SolrSuggester.init init: {name=suggest_fuzzy_file_based,lookupImpl=FuzzyLookupFactory,sourceLocation=fuzzysuggest.txt,storeDir=suggest_fuzzy_file_based,suggestAnalyzerFieldType=text,buildOnCommit=true}
   [junit4]   2> 148333 T226 oasss.SolrSuggester.init No dictionaryImpl parameter was provided falling back to org.apache.solr.spelling.suggest.FileDictionaryFactory
   [junit4]   2> 148333 T226 oasss.SolrSuggester.init Dictionary loaded with params: {name=suggest_fuzzy_file_based,lookupImpl=FuzzyLookupFactory,sourceLocation=fuzzysuggest.txt,storeDir=suggest_fuzzy_file_based,suggestAnalyzerFieldType=text,buildOnCommit=true}
   [junit4]   2> 148333 T226 oashc.SuggestComponent.inform Registering newSearcher listener for suggester: suggest_fuzzy_file_based
   [junit4]   2> 148334 T226 oasss.SolrSuggester.init init: {name=suggest_fuzzy_doc_dict,lookupImpl=FuzzyLookupFactory,dictionaryImpl=DocumentDictionaryFactory,field=cat,weightField=price,storeDir=suggest_fuzzy_doc_dict_payload,suggestAnalyzerFieldType=text,buildOnCommit=true}
   [junit4]   2> 148359 T226 oasss.SolrSuggester.init Dictionary loaded with params: {name=suggest_fuzzy_doc_dict,lookupImpl=FuzzyLookupFactory,dictionaryImpl=DocumentDictionaryFactory,field=cat,weightField=price,storeDir=suggest_fuzzy_doc_dict_payload,suggestAnalyzerFieldType=text,buildOnCommit=true}
   [junit4]   2> 148359 T226 oashc.SuggestComponent.inform Registering newSearcher listener for suggester: suggest_fuzzy_doc_dict
   [junit4]   2> 148359 T226 oasss.SolrSuggester.init init: {name=suggest_fuzzy_doc_expr_dict,dictionaryImpl=DocumentExpressionDictionaryFactory,lookupImpl=FuzzyLookupFactory,field=cat,weightExpression=((price * 2) + weight),sortField=weight,sortField=price,storeDir=suggest_fuzzy_doc_expr_dict,suggestAnalyzerFieldType=text,buildOnCommit=true}
   [junit4]   2> 148377 T226 oasss.SolrSuggester.init Dictionary loaded with params: {name=suggest_fuzzy_doc_expr_dict,dictionaryImpl=DocumentExpressionDictionaryFactory,lookupImpl=FuzzyLookupFactory,field=cat,weightExpression=((price * 2) + weight),sortField=weight,sortField=price,storeDir=suggest_fuzzy_doc_expr_dict,suggestAnalyzerFieldType=text,buildOnCommit=true}
   [junit4]   2> 148377 T226 oashc.SuggestComponent.inform Registering newSearcher listener for suggester: suggest_fuzzy_doc_expr_dict
   [junit4]   2> 148377 T226 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 148377 T227 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@18ef251[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 148378 T215 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 148378 T215 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 148380 T215 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 148394 T215 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 148396 T215 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:48458
   [junit4]   2> 148398 T215 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 148398 T215 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 148399 T215 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr
   [junit4]   2> 148399 T215 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/'
   [junit4]   2> 148418 T215 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/solr.xml
   [junit4]   2> 148473 T215 oasc.CoreContainer.<init> New CoreContainer 23269538
   [junit4]   2> 148473 T215 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/]
   [junit4]   2> 148474 T215 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 148474 T215 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: https
   [junit4]   2> 148475 T215 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 148475 T215 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 148475 T215 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 148475 T215 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 148476 T215 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 148476 T215 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 148476 T215 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 148479 T215 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 148479 T215 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 148480 T215 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 148491 T238 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1
   [junit4]   2> 148492 T238 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 148492 T238 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 148493 T238 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 148493 T238 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 148542 T238 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 148618 T238 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-suggestercomponent.xml
   [junit4]   2> 148618 T238 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 148629 T238 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 149202 T238 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 149208 T238 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 149215 T238 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 149221 T238 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 149224 T238 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 149226 T238 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 149227 T238 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 149227 T238 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 149227 T238 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 149228 T238 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 149228 T238 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 149228 T238 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
   [junit4]   2> 149228 T238 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/, dataDir=./org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/shard0/data/
   [junit4]   2> 149229 T238 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 149229 T238 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/shard0/data
   [junit4]   2> 149229 T238 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/shard0/data/index/
   [junit4]   2> 149229 T238 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/shard0/data/index' doesn't exist. Creating new index...
   [junit4]   2> 149230 T238 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/shard0/data/index
   [junit4]   2> 149230 T238 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=32, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=28.2919921875, floorSegmentMB=0.685546875, forceMergeDeletesPctAllowed=4.651501728419072, segmentsPerTier=44.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 149233 T238 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=NRTCachingDirectory(NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/shard0/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1fe6f06; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
   [junit4]   2> 149234 T238 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 149245 T238 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 149245 T238 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 149245 T238 oasc.RequestHandlers.initHandlersFromConfig created /suggest: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 149246 T238 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 149248 T238 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 149250 T238 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 149251 T238 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 149251 T238 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=50, maxMergeAtOnceExplicit=13, maxMergedSegmentMB=54.0068359375, floorSegmentMB=1.7255859375, forceMergeDeletesPctAllowed=9.61680111966303, segmentsPerTier=14.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.33166774602003857
   [junit4]   2> 149252 T238 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=NRTCachingDirectory(NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/shard0/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1fe6f06; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
   [junit4]   2> 149252 T238 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 149253 T238 oass.SolrIndexSearcher.<init> Opening Searcher@40e398[collection1] main
   [junit4]   2> 149253 T238 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf
   [junit4]   2> 149253 T238 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/}
   [junit4]   2> 149253 T238 oasr.ManagedResourceStorage.load Reading _rest_managed.json using file:dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf
   [junit4]   2> 149254 T238 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
   [junit4]   2> 149254 T238 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 149254 T238 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 149254 T238 oashc.SuggestComponent.inform Initializing SuggestComponent
   [junit4]   2> 149255 T238 oasss.SolrSuggester.init init: {name=suggest_fuzzy_with_high_freq_dict,lookupImpl=FuzzyLookupFactory,field=cat,storeDir=suggest_fuzzy_with_high_freq_dict,suggestAnalyzerFieldType=text,buildOnCommit=true,threshold=0.0}
   [junit4]   2> 149255 T238 oasss.SolrSuggester.init No dictionaryImpl parameter was provided falling back to org.apache.solr.spelling.suggest.HighFrequencyDictionaryFactory
   [junit4]   2> 149256 T238 oasss.SolrSuggester.init Dictionary loaded with params: {name=suggest_fuzzy_with_high_freq_dict,lookupImpl=FuzzyLookupFactory,field=cat,storeDir=suggest_fuzzy_with_high_freq_dict,suggestAnalyzerFieldType=text,buildOnCommit=true,threshold=0.0}
   [junit4]   2> 149256 T238 oashc.SuggestComponent.inform Registering newSearcher listener for suggester: suggest_fuzzy_with_high_freq_dict
   [junit4]   2> 149256 T238 oasss.SolrSuggester.init init: {name=suggest_fuzzy_file_based,lookupImpl=FuzzyLookupFactory,sourceLocation=fuzzysuggest.txt,storeDir=suggest_fuzzy_file_based,suggestAnalyzerFieldType=text,buildOnCommit=true}
   [junit4]   2> 149256 T238 oasss.SolrSuggester.init No dictionaryImpl parameter was provided falling back to org.apache.solr.spelling.suggest.FileDictionaryFactory
   [junit4]   2> 149257 T238 oasss.SolrSuggester.init Dictionary loaded with params: {name=suggest_fuzzy_file_based,lookupImpl=FuzzyLookupFactory,sourceLocation=fuzzysuggest.txt,storeDir=suggest_fuzzy_file_based,suggestAnalyzerFieldType=text,buildOnCommit=true}
   [junit4]   2> 149257 T238 oashc.SuggestComponent.inform Registering newSearcher listener for suggester: suggest_fuzzy_file_based
   [junit4]   2> 149257 T238 oasss.SolrSuggester.init init: {name=suggest_fuzzy_doc_dict,lookupImpl=FuzzyLookupFactory,dictionaryImpl=DocumentDictionaryFactory,field=cat,weightField=price,storeDir=suggest_fuzzy_doc_dict_payload,suggestAnalyzerFieldType=text,buildOnCommit=true}
   [junit4]   2> 149257 T238 oasss.SolrSuggester.init Dictionary loaded with params: {name=suggest_fuzzy_doc_dict,lookupImpl=FuzzyLookupFactory,dictionaryImpl=DocumentDictionaryFactory,field=cat,weightField=price,storeDir=suggest_fuzzy_doc_dict_payload,suggestAnalyzerFieldType=text,buildOnCommit=true}
   [junit4]   2> 149258 T238 oashc.SuggestComponent.inform Registering newSearcher listener for suggester: suggest_fuzzy_doc_dict
   [junit4]   2> 149258 T238 oasss.SolrSuggester.init init: {name=suggest_fuzzy_doc_expr_dict,dictionaryImpl=DocumentExpressionDictionaryFactory,lookupImpl=FuzzyLookupFactory,field=cat,weightExpression=((price * 2) + weight),sortField=weight,sortField=price,storeDir=suggest_fuzzy_doc_expr_dict,suggestAnalyzerFieldType=text,buildOnCommit=true}
   [junit4]   2> 149258 T238 oasss.SolrSuggester.init Dictionary loaded with params: {name=suggest_fuzzy_doc_expr_dict,dictionaryImpl=DocumentExpressionDictionaryFactory,lookupImpl=FuzzyLookupFactory,field=cat,weightExpression=((price * 2) + weight),sortField=weight,sortField=price,storeDir=suggest_fuzzy_doc_expr_dict,suggestAnalyzerFieldType=text,buildOnCommit=true}
   [junit4]   2> 149258 T238 oashc.SuggestComponent.inform Registering newSearcher listener for suggester: suggest_fuzzy_doc_expr_dict
   [junit4]   2> 149259 T238 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 149259 T239 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@40e398[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 149259 T215 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 149259 T215 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> ASYNC  NEW_CORE C190 name=collection1 org.apache.solr.core.SolrCore@ff4207
   [junit4]   2> 149392 T223 C190 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 149392 T223 C190 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {deleteByQuery=*:*} 0 1
   [junit4]   2> ASYNC  NEW_CORE C191 name=collection1 org.apache.solr.core.SolrCore@16e83c3
   [junit4]   2> 149462 T235 C191 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 149462 T235 C191 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {deleteByQuery=*:*} 0 1
   [junit4]   2> 149469 T222 C190 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[1]} 0 3
   [junit4]   2> 149476 T236 C191 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[1]} 0 2
   [junit4]   2> 149482 T224 C190 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[2]} 0 3
   [junit4]   2> 149486 T237 C191 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[2]} 0 1
   [junit4]   2> 149489 T225 C190 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[3]} 0 0
   [junit4]   2> 149494 T234 C191 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[3]} 0 1
   [junit4]   2> 149497 T223 C190 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[4]} 0 0
   [junit4]   2> 149501 T235 C191 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[4]} 0 1
   [junit4]   2> 149505 T222 C190 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[5]} 0 1
   [junit4]   2> 149509 T236 C191 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[5]} 0 1
   [junit4]   2> 149512 T224 C190 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[6]} 0 0
   [junit4]   2> 149516 T237 C191 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[6]} 0 1
   [junit4]   2> 149519 T225 C190 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[7]} 0 0
   [junit4]   2> 149523 T234 C191 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[7]} 0 1
   [junit4]   2> 149526 T223 C190 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[8]} 0 0
   [junit4]   2> 149530 T235 C191 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[8]} 0 1
   [junit4]   2> 149534 T222 C190 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[9]} 0 0
   [junit4]   2> 149539 T236 C191 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[9]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C192 name=collection1 org.apache.solr.core.SolrCore@ff4207
   [junit4]   2> 149547 T224 C192 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10]} 0 0
   [junit4]   2> ASYNC  NEW_CORE C193 name=collection1 org.apache.solr.core.SolrCore@16e83c3
   [junit4]   2> 149551 T237 C193 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10]} 0 1
   [junit4]   2> 149554 T225 C192 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 149587 T225 C192 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=NRTCachingDirectory(NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/control/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@25ec7; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=NRTCachingDirectory(NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/control/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@25ec7; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
   [junit4]   2> 149587 T225 C192 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 149607 T225 C192 oass.SolrIndexSearcher.<init> Opening Searcher@19a9c63[collection1] main
   [junit4]   2> 149609 T225 C192 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 149609 T227 oashc.SuggestComponent$SuggesterListener.buildSuggesterIndex Building suggester index for: suggest_fuzzy_with_high_freq_dict
   [junit4]   2> 149614 T227 oasss.SolrSuggester.build build()
   [junit4]   2> 149641 T227 oasss.SolrSuggester.build Stored suggest data to: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/control/data/suggest_fuzzy_with_high_freq_dict/fwfsta.bin
   [junit4]   2> 149642 T227 oashc.SuggestComponent$SuggesterListener.buildSuggesterIndex Building suggester index for: suggest_fuzzy_file_based
   [junit4]   2> 149642 T227 oasss.SolrSuggester.build build()
   [junit4]   2> 149648 T227 oasss.SolrSuggester.build Stored suggest data to: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/control/data/suggest_fuzzy_file_based/fwfsta.bin
   [junit4]   2> 149648 T227 oashc.SuggestComponent$SuggesterListener.buildSuggesterIndex Building suggester index for: suggest_fuzzy_doc_dict
   [junit4]   2> 149649 T227 oasss.SolrSuggester.build build()
   [junit4]   2> 149661 T227 oasss.SolrSuggester.build Stored suggest data to: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/control/data/suggest_fuzzy_doc_dict_payload/fwfsta.bin
   [junit4]   2> 149661 T227 oashc.SuggestComponent$SuggesterListener.buildSuggesterIndex Building suggester index for: suggest_fuzzy_doc_expr_dict
   [junit4]   2> 149661 T227 oasss.SolrSuggester.build build()
   [junit4]   2> 149865 T227 oasss.SolrSuggester.build Stored suggest data to: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/control/data/suggest_fuzzy_doc_expr_dict/fwfsta.bin
   [junit4]   2> 149866 T227 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@19a9c63[collection1] main{StandardDirectoryReader(segments_2:4:nrt _0(5.0):C10)}
   [junit4]   2> ASYNC  NEW_CORE C194 name=collection1 org.apache.solr.core.SolrCore@ff4207
   [junit4]   2> 149866 T225 C194 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&softCommit=false&wt=javabin&commit=true&waitSearcher=true} {commit=} 0 312
   [junit4]   2> ASYNC  NEW_CORE C195 name=collection1 org.apache.solr.core.SolrCore@16e83c3
   [junit4]   2> 149869 T235 C195 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> ASYNC  NEW_CORE C196 name=collection1 org.apache.solr.core.SolrCore@16e83c3
   [junit4]   2> 149927 T235 C196 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=NRTCachingDirectory(NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/shard0/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1fe6f06; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=NRTCachingDirectory(NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/shard0/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1fe6f06; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
   [junit4]   2> 149927 T235 C196 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 149932 T235 C196 oass.SolrIndexSearcher.<init> Opening Searcher@a68488[collection1] main
   [junit4]   2> 149933 T235 C196 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 149933 T239 oashc.SuggestComponent$SuggesterListener.buildSuggesterIndex Building suggester index for: suggest_fuzzy_with_high_freq_dict
   [junit4]   2> 149933 T239 oasss.SolrSuggester.build build()
   [junit4]   2> 149937 T239 oasss.SolrSuggester.build Stored suggest data to: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/shard0/data/suggest_fuzzy_with_high_freq_dict/fwfsta.bin
   [junit4]   2> 149937 T239 oashc.SuggestComponent$SuggesterListener.buildSuggesterIndex Building suggester index for: suggest_fuzzy_file_based
   [junit4]   2> 149938 T239 oasss.SolrSuggester.build build()
   [junit4]   2> 149940 T239 oasss.SolrSuggester.build Stored suggest data to: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/shard0/data/suggest_fuzzy_file_based/fwfsta.bin
   [junit4]   2> 149940 T239 oashc.SuggestComponent$SuggesterListener.buildSuggesterIndex Building suggester index for: suggest_fuzzy_doc_dict
   [junit4]   2> 149940 T239 oasss.SolrSuggester.build build()
   [junit4]   2> 149945 T239 oasss.SolrSuggester.build Stored suggest data to: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/shard0/data/suggest_fuzzy_doc_dict_payload/fwfsta.bin
   [junit4]   2> 149945 T239 oashc.SuggestComponent$SuggesterListener.buildSuggesterIndex Building suggester index for: suggest_fuzzy_doc_expr_dict
   [junit4]   2> 149945 T239 oasss.SolrSuggester.build build()
   [junit4]   2> 149950 T239 oasss.SolrSuggester.build Stored suggest data to: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/shard0/data/suggest_fuzzy_doc_expr_dict/fwfsta.bin
   [junit4]   2> 149950 T239 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@a68488[collection1] main{StandardDirectoryReader(segments_2:4:nrt _0(5.0):C10)}
   [junit4]   2> 149951 T235 C196 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&softCommit=false&wt=javabin&commit=true&waitSearcher=true} {commit=} 0 82
   [junit4]   2> ASYNC  NEW_CORE C197 name=collection1 org.apache.solr.core.SolrCore@ff4207
   [junit4]   2> 149954 T222 C197 oashc.SuggestComponent.prepare SuggestComponent prepare with : {params(qt=/suggest&version=2&suggest.q=&suggest.buildAll=true&shards.qt=/suggest&wt=javabin&distrib=false),defaults(suggest=true)}
   [junit4]   2> 149955 T222 C197 oasss.SolrSuggester.build build()
   [junit4]   2> 149960 T222 C197 oasss.SolrSuggester.build Stored suggest data to: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/control/data/suggest_fuzzy_with_high_freq_dict/fwfsta.bin
   [junit4]   2> 149961 T222 C197 oasss.SolrSuggester.build build()
   [junit4]   2> 149965 T222 C197 oasss.SolrSuggester.build Stored suggest data to: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/control/data/suggest_fuzzy_file_based/fwfsta.bin
   [junit4]   2> 149965 T222 C197 oasss.SolrSuggester.build build()
   [junit4]   2> 149972 T222 C197 oasss.SolrSuggester.build Stored suggest data to: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/control/data/suggest_fuzzy_doc_expr_dict/fwfsta.bin
   [junit4]   2> 149972 T222 C197 oasss.SolrSuggester.build build()
   [junit4]   2> ASYNC  NEW_CORE C198 name=collection1 org.apache.solr.core.SolrCore@ff4207
   [junit4]   2> 149981 T222 C198 oasss.SolrSuggester.build Stored suggest data to: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/control/data/suggest_fuzzy_doc_dict_payload/fwfsta.bin
   [junit4]   2> 149982 T222 C198 oashc.SuggestComponent.process SuggestComponent process with : {params(qt=/suggest&version=2&suggest.q=&suggest.buildAll=true&shards.qt=/suggest&wt=javabin&distrib=false),defaults(suggest=true)}
   [junit4]   2> 149983 T222 C198 oasc.SolrCore.execute [collection1] webapp= path=/suggest params={qt=/suggest&version=2&suggest.q=&suggest.buildAll=true&shards.qt=/suggest&wt=javabin&distrib=false} status=0 QTime=29 
   [junit4]   2> ASYNC  NEW_CORE C199 name=collection1 org.apache.solr.core.SolrCore@16e83c3
   [junit4]   2> 149986 T237 C199 oashc.SuggestComponent.prepare SuggestComponent prepare with : {params(qt=/suggest&version=2&suggest.q=&suggest.buildAll=true&shards.qt=/suggest&wt=javabin&shards=[ff01::114]:33332/|[ff01::083]:33332/|https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 149987 T237 C199 oasss.SolrSuggester.build build()
   [junit4]   2> 149990 T237 C199 oasss.SolrSuggester.build Stored suggest data to: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/shard0/data/suggest_fuzzy_with_high_freq_dict/fwfsta.bin
   [junit4]   2> 149990 T237 C199 oasss.SolrSuggester.build build()
   [junit4]   2> 149993 T237 C199 oasss.SolrSuggester.build Stored suggest data to: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/shard0/data/suggest_fuzzy_file_based/fwfsta.bin
   [junit4]   2> 149993 T237 C199 oasss.SolrSuggester.build build()
   [junit4]   2> 149998 T237 C199 oasss.SolrSuggester.build Stored suggest data to: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/shard0/data/suggest_fuzzy_doc_expr_dict/fwfsta.bin
   [junit4]   2> 149998 T237 C199 oasss.SolrSuggester.build build()
   [junit4]   2> 150001 T237 C199 oasss.SolrSuggester.build Stored suggest data to: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/shard0/data/suggest_fuzzy_doc_dict_payload/fwfsta.bin
   [junit4]   2> 150001 T237 C199 oashc.SuggestComponent.distributedProcess SuggestComponent distributedProcess with : {params(qt=/suggest&version=2&suggest.q=&suggest.buildAll=true&shards.qt=/suggest&wt=javabin&shards=[ff01::114]:33332/|[ff01::083]:33332/|https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 150002 T237 C199 oashc.SuggestComponent.finishStage SuggestComponent finishStage with : {params(qt=/suggest&version=2&suggest.q=&suggest.buildAll=true&shards.qt=/suggest&wt=javabin&shards=[ff01::114]:33332/|[ff01::083]:33332/|https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 150002 T237 C199 oashc.SuggestComponent.distributedProcess SuggestComponent distributedProcess with : {params(qt=/suggest&version=2&suggest.q=&suggest.buildAll=true&shards.qt=/suggest&wt=javabin&shards=[ff01::114]:33332/|[ff01::083]:33332/|https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 150174 T234 C199 oashc.SuggestComponent.prepare SuggestComponent prepare with : {params(NOW=1395244950446&qt=/suggest&version=2&suggest.q=&suggest.buildAll=true&shards.qt=/suggest&suggest=true&wt=javabin&shard.url=[ff01::114]:33332/|[ff01::083]:33332/|https://127.0.0.1:48458/&distrib=false&isShard=true),defaults(suggest=true)}
   [junit4]   2> 150175 T234 C199 oasss.SolrSuggester.build build()
   [junit4]   2> 150179 T234 C199 oasss.SolrSuggester.build Stored suggest data to: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/shard0/data/suggest_fuzzy_with_high_freq_dict/fwfsta.bin
   [junit4]   2> 150180 T234 C199 oasss.SolrSuggester.build build()
   [junit4]   2> 150183 T234 C199 oasss.SolrSuggester.build Stored suggest data to: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/shard0/data/suggest_fuzzy_file_based/fwfsta.bin
   [junit4]   2> 150184 T234 C199 oasss.SolrSuggester.build build()
   [junit4]   2> 150189 T234 C199 oasss.SolrSuggester.build Stored suggest data to: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/shard0/data/suggest_fuzzy_doc_expr_dict/fwfsta.bin
   [junit4]   2> 150190 T234 C199 oasss.SolrSuggester.build build()
   [junit4]   2> 150194 T234 C199 oasss.SolrSuggester.build Stored suggest data to: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/shard0/data/suggest_fuzzy_doc_dict_payload/fwfsta.bin
   [junit4]   2> 150195 T234 C199 oashc.SuggestComponent.process SuggestComponent process with : {params(NOW=1395244950446&qt=/suggest&version=2&suggest.q=&suggest.buildAll=true&shards.qt=/suggest&suggest=true&wt=javabin&shard.url=[ff01::114]:33332/|[ff01::083]:33332/|https://127.0.0.1:48458/&distrib=false&isShard=true),defaults(suggest=true)}
   [junit4]   2> 150195 T234 C199 oasc.SolrCore.execute [collection1] webapp= path=/suggest params={NOW=1395244950446&qt=/suggest&version=2&suggest.q=&suggest.buildAll=true&shards.qt=/suggest&suggest=true&wt=javabin&shard.url=[ff01::114]:33332/|[ff01::083]:33332/|https://127.0.0.1:48458/&distrib=false&isShard=true} status=0 QTime=21 
   [junit4]   2> 150196 T237 C199 oashc.SuggestComponent.finishStage SuggestComponent finishStage with : {params(qt=/suggest&version=2&suggest.q=&suggest.buildAll=true&shards.qt=/suggest&wt=javabin&shards=[ff01::114]:33332/|[ff01::083]:33332/|https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 150197 T237 C199 oashc.SuggestComponent.distributedProcess SuggestComponent distributedProcess with : {params(qt=/suggest&version=2&suggest.q=&suggest.buildAll=true&shards.qt=/suggest&wt=javabin&shards=[ff01::114]:33332/|[ff01::083]:33332/|https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 150197 T237 C199 oashc.SuggestComponent.finishStage SuggestComponent finishStage with : {params(qt=/suggest&version=2&suggest.q=&suggest.buildAll=true&shards.qt=/suggest&wt=javabin&shards=[ff01::114]:33332/|[ff01::083]:33332/|https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 150197 T237 C199 oashc.SuggestComponent.finishStage [ff01::114]:33332/|[ff01::083]:33332/|https://127.0.0.1:48458/ : {}
   [junit4]   2> 150197 T237 C199 oasc.SolrCore.execute [collection1] webapp= path=/suggest params={qt=/suggest&version=2&suggest.q=&suggest.buildAll=true&shards.qt=/suggest&wt=javabin&shards=[ff01::114]:33332/|[ff01::083]:33332/|https://127.0.0.1:48458/} status=0 QTime=211 
   [junit4]   2> 150201 T225 C198 oashc.SuggestComponent.prepare SuggestComponent prepare with : {params(qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_dict&shards.qt=/suggest&wt=javabin&distrib=false&suggest.count=2&q=exampel),defaults(suggest=true)}
   [junit4]   2> 150201 T225 C198 oashc.SuggestComponent.process SuggestComponent process with : {params(qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_dict&shards.qt=/suggest&wt=javabin&distrib=false&suggest.count=2&q=exampel),defaults(suggest=true)}
   [junit4]   2> ASYNC  NEW_CORE C200 name=collection1 org.apache.solr.core.SolrCore@ff4207
   [junit4]   2> 150220 T225 C200 oasc.SolrCore.execute [collection1] webapp= path=/suggest params={qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_dict&shards.qt=/suggest&wt=javabin&distrib=false&suggest.count=2&q=exampel} status=0 QTime=19 
   [junit4]   2> ASYNC  NEW_CORE C201 name=collection1 org.apache.solr.core.SolrCore@16e83c3
   [junit4]   2> 150223 T236 C201 oashc.SuggestComponent.prepare SuggestComponent prepare with : {params(qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&q=exampel&shards=[ff01::114]:33332/|https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 150223 T236 C201 oashc.SuggestComponent.distributedProcess SuggestComponent distributedProcess with : {params(qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&q=exampel&shards=[ff01::114]:33332/|https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 150224 T236 C201 oashc.SuggestComponent.finishStage SuggestComponent finishStage with : {params(qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&q=exampel&shards=[ff01::114]:33332/|https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 150224 T236 C201 oashc.SuggestComponent.distributedProcess SuggestComponent distributedProcess with : {params(qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&q=exampel&shards=[ff01::114]:33332/|https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 150227 T235 C201 oashc.SuggestComponent.prepare SuggestComponent prepare with : {params(NOW=1395244950683&qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_dict&shards.qt=/suggest&suggest=true&wt=javabin&shard.url=[ff01::114]:33332/|https://127.0.0.1:48458/&distrib=false&suggest.count=2&isShard=true&q=exampel),defaults(suggest=true)}
   [junit4]   2> 150227 T235 C201 oashc.SuggestComponent.process SuggestComponent process with : {params(NOW=1395244950683&qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_dict&shards.qt=/suggest&suggest=true&wt=javabin&shard.url=[ff01::114]:33332/|https://127.0.0.1:48458/&distrib=false&suggest.count=2&isShard=true&q=exampel),defaults(suggest=true)}
   [junit4]   2> 150228 T235 C201 oasc.SolrCore.execute [collection1] webapp= path=/suggest params={NOW=1395244950683&qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_dict&shards.qt=/suggest&suggest=true&wt=javabin&shard.url=[ff01::114]:33332/|https://127.0.0.1:48458/&distrib=false&suggest.count=2&isShard=true&q=exampel} status=0 QTime=1 
   [junit4]   2> 150229 T236 C201 oashc.SuggestComponent.finishStage SuggestComponent finishStage with : {params(qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&q=exampel&shards=[ff01::114]:33332/|https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 150229 T236 C201 oashc.SuggestComponent.distributedProcess SuggestComponent distributedProcess with : {params(qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&q=exampel&shards=[ff01::114]:33332/|https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 150230 T236 C201 oashc.SuggestComponent.finishStage SuggestComponent finishStage with : {params(qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&q=exampel&shards=[ff01::114]:33332/|https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 150230 T236 C201 oashc.SuggestComponent.finishStage [ff01::114]:33332/|https://127.0.0.1:48458/ : {suggest_fuzzy_doc_dict={exampel={numFound=2,suggestions=[{term=example inputdata,weight=45,payload=}, {term=example data,weight=40,payload=}]}}}
   [junit4]   2> 150232 T236 C201 oasc.SolrCore.execute [collection1] webapp= path=/suggest params={qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&q=exampel&shards=[ff01::114]:33332/|https://127.0.0.1:48458/} status=0 QTime=9 
   [junit4]   2> 150235 T223 C200 oashc.SuggestComponent.prepare SuggestComponent prepare with : {params(qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&wt=javabin&distrib=false&suggest.count=2&q=Yet),defaults(suggest=true)}
   [junit4]   2> 150236 T223 C200 oashc.SuggestComponent.process SuggestComponent process with : {params(qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&wt=javabin&distrib=false&suggest.count=2&q=Yet),defaults(suggest=true)}
   [junit4]   2> 150237 T223 C200 oasc.SolrCore.execute [collection1] webapp= path=/suggest params={qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&wt=javabin&distrib=false&suggest.count=2&q=Yet} status=0 QTime=2 
   [junit4]   2> 150240 T234 C201 oashc.SuggestComponent.prepare SuggestComponent prepare with : {params(qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&q=Yet&shards=[ff01::083]:33332/|https://127.0.0.1:48458/|[ff01::213]:33332/),defaults(suggest=true)}
   [junit4]   2> 150241 T234 C201 oashc.SuggestComponent.distributedProcess SuggestComponent distributedProcess with : {params(qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&q=Yet&shards=[ff01::083]:33332/|https://127.0.0.1:48458/|[ff01::213]:33332/),defaults(suggest=true)}
   [junit4]   2> 150241 T234 C201 oashc.SuggestComponent.finishStage SuggestComponent finishStage with : {params(qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&q=Yet&shards=[ff01::083]:33332/|https://127.0.0.1:48458/|[ff01::213]:33332/),defaults(suggest=true)}
   [junit4]   2> 150241 T234 C201 oashc.SuggestComponent.distributedProcess SuggestComponent distributedProcess with : {params(qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&q=Yet&shards=[ff01::083]:33332/|https://127.0.0.1:48458/|[ff01::213]:33332/),defaults(suggest=true)}
   [junit4]   2> 155252 T236 C201 oashc.SuggestComponent.prepare SuggestComponent prepare with : {params(NOW=1395244950700&qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&suggest=true&wt=javabin&shard.url=[ff01::083]:33332/|https://127.0.0.1:48458/|[ff01::213]:33332/&distrib=false&suggest.count=2&isShard=true&q=Yet),defaults(suggest=true)}
   [junit4]   2> 155252 T236 C201 oashc.SuggestComponent.process SuggestComponent process with : {params(NOW=1395244950700&qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&suggest=true&wt=javabin&shard.url=[ff01::083]:33332/|https://127.0.0.1:48458/|[ff01::213]:33332/&distrib=false&suggest.count=2&isShard=true&q=Yet),defaults(suggest=true)}
   [junit4]   2> 155254 T236 C201 oasc.SolrCore.execute [collection1] webapp= path=/suggest params={NOW=1395244950700&qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&suggest=true&wt=javabin&shard.url=[ff01::083]:33332/|https://127.0.0.1:48458/|[ff01::213]:33332/&distrib=false&suggest.count=2&isShard=true&q=Yet} status=0 QTime=3 
   [junit4]   2> 155256 T234 C201 oashc.SuggestComponent.finishStage SuggestComponent finishStage with : {params(qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&q=Yet&shards=[ff01::083]:33332/|https://127.0.0.1:48458/|[ff01::213]:33332/),defaults(suggest=true)}
   [junit4]   2> 155256 T234 C201 oashc.SuggestComponent.distributedProcess SuggestComponent distributedProcess with : {params(qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&q=Yet&shards=[ff01::083]:33332/|https://127.0.0.1:48458/|[ff01::213]:33332/),defaults(suggest=true)}
   [junit4]   2> 155257 T234 C201 oashc.SuggestComponent.finishStage SuggestComponent finishStage with : {params(qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&q=Yet&shards=[ff01::083]:33332/|https://127.0.0.1:48458/|[ff01::213]:33332/),defaults(suggest=true)}
   [junit4]   2> 155257 T234 C201 oashc.SuggestComponent.finishStage [ff01::083]:33332/|https://127.0.0.1:48458/|[ff01::213]:33332/ : {suggest_fuzzy_doc_expr_dict={Yet={numFound=2,suggestions=[{term=Yet another title,weight=60,payload=}, {term=Yet another,weight=40,payload=}]}}}
   [junit4]   2> 155258 T234 C201 oasc.SolrCore.execute [collection1] webapp= path=/suggest params={qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&q=Yet&shards=[ff01::083]:33332/|https://127.0.0.1:48458/|[ff01::213]:33332/} status=0 QTime=5018 
   [junit4]   2> 155263 T224 C200 oashc.SuggestComponent.prepare SuggestComponent prepare with : {params(qt=/suggest&version=2&suggest.q=blah&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&wt=javabin&distrib=false&suggest.count=2),defaults(suggest=true)}
   [junit4]   2> 155263 T224 C200 oashc.SuggestComponent.process SuggestComponent process with : {params(qt=/suggest&version=2&suggest.q=blah&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&wt=javabin&distrib=false&suggest.count=2),defaults(suggest=true)}
   [junit4]   2> 155265 T224 C200 oasc.SolrCore.execute [collection1] webapp= path=/suggest params={qt=/suggest&version=2&suggest.q=blah&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&wt=javabin&distrib=false&suggest.count=2} status=0 QTime=3 
   [junit4]   2> 155269 T235 C201 oashc.SuggestComponent.prepare SuggestComponent prepare with : {params(qt=/suggest&version=2&suggest.q=blah&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&shards=https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 155269 T235 C201 oashc.SuggestComponent.distributedProcess SuggestComponent distributedProcess with : {params(qt=/suggest&version=2&suggest.q=blah&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&shards=https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 155270 T235 C201 oashc.SuggestComponent.finishStage SuggestComponent finishStage with : {params(qt=/suggest&version=2&suggest.q=blah&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&shards=https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 155270 T235 C201 oashc.SuggestComponent.distributedProcess SuggestComponent distributedProcess with : {params(qt=/suggest&version=2&suggest.q=blah&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&shards=https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 155274 T237 C201 oashc.SuggestComponent.prepare SuggestComponent prepare with : {params(NOW=1395244955728&qt=/suggest&version=2&suggest.q=blah&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&suggest=true&wt=javabin&shard.url=https://127.0.0.1:48458/&distrib=false&suggest.count=2&isShard=true),defaults(suggest=true)}
   [junit4]   2> 155274 T237 C201 oashc.SuggestComponent.process SuggestComponent process with : {params(NOW=1395244955728&qt=/suggest&version=2&suggest.q=blah&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&suggest=true&wt=javabin&shard.url=https://127.0.0.1:48458/&distrib=false&suggest.count=2&isShard=true),defaults(suggest=true)}
   [junit4]   2> 155275 T237 C201 oasc.SolrCore.execute [collection1] webapp= path=/suggest params={NOW=1395244955728&qt=/suggest&version=2&suggest.q=blah&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&suggest=true&wt=javabin&shard.url=https://127.0.0.1:48458/&distrib=false&suggest.count=2&isShard=true} status=0 QTime=2 
   [junit4]   2> 155277 T235 C201 oashc.SuggestComponent.finishStage SuggestComponent finishStage with : {params(qt=/suggest&version=2&suggest.q=blah&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&shards=https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 155278 T235 C201 oashc.SuggestComponent.distributedProcess SuggestComponent distributedProcess with : {params(qt=/suggest&version=2&suggest.q=blah&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&shards=https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 155278 T235 C201 oashc.SuggestComponent.finishStage SuggestComponent finishStage with : {params(qt=/suggest&version=2&suggest.q=blah&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&shards=https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 155279 T235 C201 oashc.SuggestComponent.finishStage https://127.0.0.1:48458/ : {suggest_fuzzy_doc_expr_dict={blah={numFound=1,suggestions=[{term=blah in blah,weight=140,payload=}]}}}
   [junit4]   2> 155279 T235 C201 oasc.SolrCore.execute [collection1] webapp= path=/suggest params={qt=/suggest&version=2&suggest.q=blah&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&shards=https://127.0.0.1:48458/} status=0 QTime=11 
   [junit4]   2> 155283 T223 C200 oashc.SuggestComponent.prepare SuggestComponent prepare with : {params(qt=/suggest&version=2&suggest.q=blah&suggest.dictionary=suggest_fuzzy_doc_dict&shards.qt=/suggest&wt=javabin&distrib=false&suggest.count=2),defaults(suggest=true)}
   [junit4]   2> 155284 T223 C200 oashc.SuggestComponent.process SuggestComponent process with : {params(qt=/suggest&version=2&suggest.q=blah&suggest.dictionary=suggest_fuzzy_doc_dict&shards.qt=/suggest&wt=javabin&distrib=false&suggest.count=2),defaults(suggest=true)}
   [junit4]   2> 155285 T223 C200 oasc.SolrCore.execute [collection1] webapp= path=/suggest params={qt=/suggest&version=2&suggest.q=blah&suggest.dictionary=suggest_fuzzy_doc_dict&shards.qt=/suggest&wt=javabin&distrib=false&suggest.count=2} status=0 QTime=2 
   [junit4]   2> 155290 T236 C201 oashc.SuggestComponent.prepare SuggestComponent prepare with : {params(qt=/suggest&version=2&suggest.q=blah&suggest.dictionary=suggest_fuzzy_doc_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&shards=[ff01::114]:33332/|[ff01::083]:33332/|https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 155290 T236 C201 oashc.SuggestComponent.distributedProcess SuggestComponent distributedProcess with : {params(qt=/suggest&version=2&suggest.q=blah&suggest.dictionary=suggest_fuzzy_doc_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&shards=[ff01::114]:33332/|[ff01::083]:33332/|https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 155291 T236 C201 oashc.SuggestComponent.finishStage SuggestComponent finishStage with : {params(qt=/suggest&version=2&suggest.q=blah&suggest.dictionary=suggest_fuzzy_doc_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&shards=[ff01::114]:33332/|[ff01::083]:33332/|https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 155291 T236 C201 oashc.SuggestComponent.distributedProcess SuggestComponent distributedProcess with : {params(qt=/suggest&version=2&suggest.q=blah&suggest.dictionary=suggest_fuzzy_doc_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&shards=[ff01::114]:33332/|[ff01::083]:33332/|https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 155294 T234 C201 oashc.SuggestComponent.prepare SuggestComponent prepare with : {params(NOW=1395244955749&qt=/suggest&version=2&suggest.q=blah&suggest.dictionary=suggest_fuzzy_doc_dict&shards.qt=/suggest&suggest=true&wt=javabin&shard.url=[ff01::114]:33332/|[ff01::083]:33332/|https://127.0.0.1:48458/&distrib=false&suggest.count=2&isShard=true),defaults(suggest=true)}
   [junit4]   2> 155295 T234 C201 oashc.SuggestComponent.process SuggestComponent process with : {params(NOW=1395244955749&qt=/suggest&version=2&suggest.q=blah&suggest.dictionary=suggest_fuzzy_doc_dict&shards.qt=/suggest&suggest=true&wt=javabin&shard.url=[ff01::114]:33332/|[ff01::083]:33332/|https://127.0.0.1:48458/&distrib=false&suggest.count=2&isShard=true),defaults(suggest=true)}
   [junit4]   2> 155296 T234 C201 oasc.SolrCore.execute [collection1] webapp= path=/suggest params={NOW=1395244955749&qt=/suggest&version=2&suggest.q=blah&suggest.dictionary=suggest_fuzzy_doc_dict&shards.qt=/suggest&suggest=true&wt=javabin&shard.url=[ff01::114]:33332/|[ff01::083]:33332/|https://127.0.0.1:48458/&distrib=false&suggest.count=2&isShard=true} status=0 QTime=2 
   [junit4]   2> 155298 T236 C201 oashc.SuggestComponent.finishStage SuggestComponent finishStage with : {params(qt=/suggest&version=2&suggest.q=blah&suggest.dictionary=suggest_fuzzy_doc_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&shards=[ff01::114]:33332/|[ff01::083]:33332/|https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 155299 T236 C201 oashc.SuggestComponent.distributedProcess SuggestComponent distributedProcess with : {params(qt=/suggest&version=2&suggest.q=blah&suggest.dictionary=suggest_fuzzy_doc_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&shards=[ff01::114]:33332/|[ff01::083]:33332/|https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 155299 T236 C201 oashc.SuggestComponent.finishStage SuggestComponent finishStage with : {params(qt=/suggest&version=2&suggest.q=blah&suggest.dictionary=suggest_fuzzy_doc_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&shards=[ff01::114]:33332/|[ff01::083]:33332/|https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 155300 T236 C201 oashc.SuggestComponent.finishStage [ff01::114]:33332/|[ff01::083]:33332/|https://127.0.0.1:48458/ : {suggest_fuzzy_doc_dict={blah={numFound=1,suggestions=[{term=blah in blah,weight=50,payload=}]}}}
   [junit4]   2> 155300 T236 C201 oasc.SolrCore.execute [collection1] webapp= path=/suggest params={qt=/suggest&version=2&suggest.q=blah&suggest.dictionary=suggest_fuzzy_doc_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&shards=[ff01::114]:33332/|[ff01::083]:33332/|https://127.0.0.1:48458/} status=0 QTime=11 
   [junit4]   2> 155304 T222 C200 oashc.SuggestComponent.prepare SuggestComponent prepare with : {params(qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_dict&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&wt=javabin&distrib=false&suggest.count=2&q=exampel),defaults(suggest=true)}
   [junit4]   2> 155305 T222 C200 oashc.SuggestComponent.process SuggestComponent process with : {params(qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_dict&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&wt=javabin&distrib=false&suggest.count=2&q=exampel),defaults(suggest=true)}
   [junit4]   2> ASYNC  NEW_CORE C202 name=collection1 org.apache.solr.core.SolrCore@ff4207
   [junit4]   2> 155317 T222 C202 oasc.SolrCore.execute [collection1] webapp= path=/suggest params={qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_dict&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&wt=javabin&distrib=false&suggest.count=2&q=exampel} status=0 QTime=13 
   [junit4]   2> ASYNC  NEW_CORE C203 name=collection1 org.apache.solr.core.SolrCore@16e83c3
   [junit4]   2> 155321 T237 C203 oashc.SuggestComponent.prepare SuggestComponent prepare with : {params(qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_dict&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&q=exampel&shards=https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 155322 T237 C203 oashc.SuggestComponent.distributedProcess SuggestComponent distributedProcess with : {params(qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_dict&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&q=exampel&shards=https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 155322 T237 C203 oashc.SuggestComponent.finishStage SuggestComponent finishStage with : {params(qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_dict&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&q=exampel&shards=https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 155323 T237 C203 oashc.SuggestComponent.distributedProcess SuggestComponent distributedProcess with : {params(qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_dict&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&q=exampel&shards=https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 155326 T235 C203 oashc.SuggestComponent.prepare SuggestComponent prepare with : {params(NOW=1395244955781&qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_dict&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&suggest=true&wt=javabin&shard.url=https://127.0.0.1:48458/&distrib=false&suggest.count=2&isShard=true&q=exampel),defaults(suggest=true)}
   [junit4]   2> 155327 T235 C203 oashc.SuggestComponent.process SuggestComponent process with : {params(NOW=1395244955781&qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_dict&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&suggest=true&wt=javabin&shard.url=https://127.0.0.1:48458/&distrib=false&suggest.count=2&isShard=true&q=exampel),defaults(suggest=true)}
   [junit4]   2> 155329 T235 C203 oasc.SolrCore.execute [collection1] webapp= path=/suggest params={NOW=1395244955781&qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_dict&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&suggest=true&wt=javabin&shard.url=https://127.0.0.1:48458/&distrib=false&suggest.count=2&isShard=true&q=exampel} status=0 QTime=3 
   [junit4]   2> 155331 T237 C203 oashc.SuggestComponent.finishStage SuggestComponent finishStage with : {params(qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_dict&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&q=exampel&shards=https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 155331 T237 C203 oashc.SuggestComponent.distributedProcess SuggestComponent distributedProcess with : {params(qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_dict&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&q=exampel&shards=https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 155332 T237 C203 oashc.SuggestComponent.finishStage SuggestComponent finishStage with : {params(qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_dict&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&q=exampel&shards=https://127.0.0.1:48458/),defaults(suggest=true)}
   [junit4]   2> 155333 T237 C203 oashc.SuggestComponent.finishStage https://127.0.0.1:48458/ : {suggest_fuzzy_doc_dict={exampel={numFound=2,suggestions=[{term=example inputdata,weight=45,payload=}, {term=example data,weight=40,payload=}]}}, suggest_fuzzy_doc_expr_dict={exampel={numFound=2,suggestions=[{term=example inputdata,weight=120,payload=}, {term=example data,weight=110,payload=}]}}}
   [junit4]   2> 155333 T237 C203 oasc.SolrCore.execute [collection1] webapp= path=/suggest params={qt=/suggest&version=2&suggest.dictionary=suggest_fuzzy_doc_dict&suggest.dictionary=suggest_fuzzy_doc_expr_dict&shards.qt=/suggest&wt=javabin&suggest.count=2&q=exampel&shards=https://127.0.0.1:48458/} status=0 QTime=12 
   [junit4]   2> 155357 T215 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=3360953
   [junit4]   2> 155358 T215 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@ff4207
   [junit4]   2> 155359 T215 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=10,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
   [junit4]   2> 155360 T215 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 155361 T215 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 155361 T215 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 155363 T215 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 155366 T215 oasc.CachingDirectoryFactory.close Closing NRTCachingDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 155367 T215 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/control/data/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/control/data/index;done=false>>]
   [junit4]   2> 155367 T215 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/control/data/index
   [junit4]   2> 155368 T215 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/control/data [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/control/data;done=false>>]
   [junit4]   2> 155369 T215 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/control/data
   [junit4]   2> 155370 T215 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 155444 T215 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=23269538
   [junit4]   2> 155446 T215 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@16e83c3
   [junit4]   2> 155447 T215 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=10,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
   [junit4]   2> 155448 T215 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 155449 T215 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 155450 T215 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 155453 T215 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 155454 T215 oasc.CachingDirectoryFactory.close Closing NRTCachingDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 155455 T215 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/shard0/data [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/shard0/data;done=false>>]
   [junit4]   2> 155455 T215 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/shard0/data
   [junit4]   2> 155456 T215 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSuggestComponentTest-1395244947666/shard0/dat

[...truncated too long message...]

ttpSolrServer.executeMethod(HttpSolrServer.java:504)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:203)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
   [junit4]    > 	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.queryServer(BaseDistributedSearchTestCase.java:515)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:563)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:545)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:524)
   [junit4]    > 	at org.apache.solr.handler.component.DistributedQueryComponentCustomSortTest.doTest(DistributedQueryComponentCustomSortTest.java:70)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 2286133 T6329 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 2286133 T6329 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=9003707
   [junit4]   2> 2286133 T6329 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@145e311
   [junit4]   2> 2286135 T6329 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
   [junit4]   2> 2286136 T6329 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 2286136 T6329 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 2286137 T6329 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 2286138 T6329 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 2286138 T6329 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 2286139 T6329 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-DistributedQueryComponentCustomSortTest-1395247038930 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-DistributedQueryComponentCustomSortTest-1395247038930;done=false>>]
   [junit4]   2> 2286139 T6329 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-DistributedQueryComponentCustomSortTest-1395247038930
   [junit4]   2> 2286140 T6329 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-DistributedQueryComponentCustomSortTest-1395247038930/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-DistributedQueryComponentCustomSortTest-1395247038930/index;done=false>>]
   [junit4]   2> 2286140 T6329 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-DistributedQueryComponentCustomSortTest-1395247038930/index
   [junit4]   2> NOTE: test params are: codec=Lucene46: {text=PostingsFormat(name=Lucene41VarGapFixedInterval), id=MockVariableIntBlock(baseBlockSize=26), _version_=PostingsFormat(name=SimpleText), asdfasdf_field_should_match_catchall_dynamic_field_adsfasdf=PostingsFormat(name=Asserting)}, docValues:{payload=DocValuesFormat(name=Asserting)}, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=et_EE, timezone=Asia/Ulan_Bator
   [junit4]   2> NOTE: Linux 3.8.0-37-generic i386/Oracle Corporation 1.7.0_60-ea (32-bit)/cpus=8,threads=1,free=123562088,total=266391552
   [junit4]   2> NOTE: All tests run in this JVM: [BinaryUpdateRequestHandlerTest, SchemaVersionSpecificBehaviorTest, ExternalFileFieldSortTest, StatelessScriptUpdateProcessorFactoryTest, DirectUpdateHandlerOptimizeTest, RecoveryZkTest, ZkControllerTest, EchoParamsTest, DistributedSuggestComponentTest, SpatialFilterTest, DocValuesTest, SyncSliceTest, RangeFacetTest, StressHdfsTest, SolrCoreTest, ShardSplitTest, TestLuceneMatchVersion, TestPostingsSolrHighlighter, TestGroupingSearch, TestTrie, CachingDirectoryFactoryTest, TestFaceting, TestSolrDeletionPolicy1, TestExtendedDismaxParser, ShardRoutingTest, TermVectorComponentTest, MoreLikeThisHandlerTest, TestAnalyzeInfixSuggestions, TestSurroundQueryParser, ScriptEngineTest, TriLevelCompositeIdRoutingTest, TestSolrQueryParser, DistributedSpellCheckComponentTest, CursorMarkTest, TestSchemaVersionResource, ReturnFieldsTest, TestElisionMultitermQuery, TestFieldTypeResource, QueryElevationComponentTest, OverseerRolesTest, AutoCommitTest, BasicDistributedZkTest, TestPHPSerializedResponseWriter, SoftAutoCommitTest, DateMathParserTest, TestCollapseQParserPlugin, TestConfig, TestInfoStreamLogging, SpellPossibilityIteratorTest, TestDistributedMissingSort, TestValueSourceCache, UUIDFieldTest, TestCopyFieldCollectionResource, RegexBoostProcessorTest, TestStressVersions, DistributedExpandComponentTest, TestSchemaSimilarityResource, BadComponentTest, CSVRequestHandlerTest, TestManagedResourceStorage, TestLFUCache, TestFreeTextSuggestions, LoggingHandlerTest, AbstractAnalyticsFacetTest, ClusterStateUpdateTest, TestFileDictionaryLookup, SliceStateTest, TestCoreContainer, TestDocumentBuilder, TestInitQParser, BlockCacheTest, TestRealTimeGet, ParsingFieldUpdateProcessorsTest, CoreAdminRequestStatusTest, SynonymTokenizerTest, TestBadConfig, TestReplicationHandler, SampleTest, DocumentBuilderTest, TestJmxMonitoredMap, SearchHandlerTest, ResourceLoaderTest, PrimitiveFieldTypeTest, CoreAdminHandlerTest, TestReloadAndDeleteDocs, FunctionTest, IndexSchemaTest, DistributedQueryComponentOptimizationTest, UnloadDistributedZkTest, TestLRUCache, MigrateRouteKeyTest, LeaderElectionTest, ClusterStateTest, BasicZkTest, FullSolrCloudDistribCmdsTest, TestRandomDVFaceting, ShardRoutingCustomTest, SliceStateUpdateTest, TestDistributedSearch, TestHashPartitioner, TestJoin, TestSort, TestFunctionQuery, BasicFunctionalityTest, OverseerCollectionProcessorTest, TestIndexSearcher, ShowFileRequestHandlerTest, CurrencyFieldOpenExchangeTest, DistributedQueryElevationComponentTest, SolrIndexSplitterTest, TestCoreDiscovery, AnalysisAfterCoreReloadTest, SpellCheckCollatorTest, SuggesterFSTTest, TestStressLucene, PolyFieldTest, SolrCoreCheckLockOnStartupTest, TestUpdate, FieldMutatingUpdateProcessorTest, TestWordDelimiterFilterFactory, DefaultValueUpdateProcessorTest, SortByFunctionTest, DistanceFunctionTest, DebugComponentTest, LukeRequestHandlerTest, TestWriterPerf, DocumentAnalysisRequestHandlerTest, FileBasedSpellCheckerTest, XmlUpdateRequestHandlerTest, RequiredFieldsTest, IndexSchemaRuntimeFieldTest, UpdateRequestProcessorFactoryTest, MBeansHandlerTest, UniqFieldsUpdateProcessorFactoryTest, TestPartialUpdateDeduplication, CoreContainerCoreInitFailuresTest, HighlighterConfigTest, UpdateParamsTest, TestSolrIndexConfig, ResponseLogComponentTest, TestDocSet, TestSearchPerf, NotRequiredUniqueKeyTest, TestCharFilters, TestPerFieldSimilarity, TestLMJelinekMercerSimilarityFactory, TestIBSimilarityFactory, TestDefaultSimilarityFactory, LegacyHTMLStripCharFilterTest, URLClassifyProcessorTest, TestHighlightDedupGrouping, AbstractAnalyticsStatsTest, NoFacetTest, ExpressionTest, FieldFacetExtrasTest, QueryFacetTest, AsyncMigrateRouteKeyTest, CustomCollectionTest, DistribCursorPagingTest, TestDistribDocBasedVersion, TestModifyConfFiles, HdfsBasicDistributedZk2Test, HdfsBasicDistributedZkTest, HdfsChaosMonkeySafeLeaderTest, HdfsCollectionsAPIDistributedZkTest, HdfsSyncSliceTest, HdfsUnloadDistributedZkTest, TestImplicitCoreProperties, TestNonNRTOpen, TestSolrXml, TestSolrXmlPersistor, CoreMergeIndexesAdminHandlerTest, DistributedQueryComponentCustomSortTest]
   [junit4] Completed on J0 in 47.70s, 1 test, 1 error <<< FAILURES!

[...truncated 77 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1275: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 391 suites, 1644 tests, 2 errors, 49 ignored (28 assumptions)

Total time: 70 minutes 24 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_60-ea-b10 -client -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_51) - Build # 9847 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9847/
Java: 64bit/jdk1.7.0_51 -XX:+UseCompressedOops -XX:+UseSerialGC -XX:-UseSuperWord

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestRequestStatusCollectionAPI

Error Message:
ERROR: SolrZkClient opens=4 closes=3

Stack Trace:
java.lang.AssertionError: ERROR: SolrZkClient opens=4 closes=3
	at __randomizedtesting.SeedInfo.seed([A01EA8B7EDDF9883]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:435)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:180)
	at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:789)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestRequestStatusCollectionAPI

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.TestRequestStatusCollectionAPI:     1) Thread[id=6555, name=TEST-TestRequestStatusCollectionAPI.testDistribSearch-seed#[A01EA8B7EDDF9883]-SendThread(localhost.localdomain:43654), state=TIMED_WAITING, group=TGRP-TestRequestStatusCollectionAPI]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)    2) Thread[id=6556, name=TEST-TestRequestStatusCollectionAPI.testDistribSearch-seed#[A01EA8B7EDDF9883]-EventThread, state=WAITING, group=TGRP-TestRequestStatusCollectionAPI]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.TestRequestStatusCollectionAPI: 
   1) Thread[id=6555, name=TEST-TestRequestStatusCollectionAPI.testDistribSearch-seed#[A01EA8B7EDDF9883]-SendThread(localhost.localdomain:43654), state=TIMED_WAITING, group=TGRP-TestRequestStatusCollectionAPI]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   2) Thread[id=6556, name=TEST-TestRequestStatusCollectionAPI.testDistribSearch-seed#[A01EA8B7EDDF9883]-EventThread, state=WAITING, group=TGRP-TestRequestStatusCollectionAPI]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
	at __randomizedtesting.SeedInfo.seed([A01EA8B7EDDF9883]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestRequestStatusCollectionAPI

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=6555, name=TEST-TestRequestStatusCollectionAPI.testDistribSearch-seed#[A01EA8B7EDDF9883]-SendThread(localhost.localdomain:43654), state=TIMED_WAITING, group=TGRP-TestRequestStatusCollectionAPI]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=6555, name=TEST-TestRequestStatusCollectionAPI.testDistribSearch-seed#[A01EA8B7EDDF9883]-SendThread(localhost.localdomain:43654), state=TIMED_WAITING, group=TGRP-TestRequestStatusCollectionAPI]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
	at __randomizedtesting.SeedInfo.seed([A01EA8B7EDDF9883]:0)


REGRESSION:  org.apache.solr.cloud.TestRequestStatusCollectionAPI.testDistribSearch

Error Message:


Stack Trace:
org.apache.solr.common.cloud.ZooKeeperException: 
	at __randomizedtesting.SeedInfo.seed([A01EA8B7EDDF9883:21F826AF9A80F8BF]:0)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.connect(CloudSolrServer.java:249)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.initCloud(AbstractFullDistribZkTestBase.java:256)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:312)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:868)
	at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:826)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:862)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /live_nodes
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:259)
	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:256)
	at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:299)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.connect(CloudSolrServer.java:242)
	... 41 more




Build Log:
[...truncated 11008 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestRequestStatusCollectionAPI
   [junit4]   2> 1438887 T6534 oas.SolrTestCaseJ4.startTrackingSearchers WARN startTrackingSearchers: numOpens=6 numCloses=6
   [junit4]   2> 1438887 T6534 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /_iev/da
   [junit4]   2> 1438892 T6534 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestRequestStatusCollectionAPI-1395241343634
   [junit4]   2> 1438892 T6534 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1438893 T6535 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1438993 T6534 oasc.ZkTestServer.run start zk server on port:43654
   [junit4]   2> 1438994 T6534 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1438999 T6541 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@361199a4 name:ZooKeeperConnection Watcher:127.0.0.1:43654 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1438999 T6534 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1438999 T6534 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1439006 T6534 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1439008 T6543 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2d077750 name:ZooKeeperConnection Watcher:127.0.0.1:43654/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1439008 T6534 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1439008 T6534 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1439012 T6534 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1439016 T6534 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1439019 T6534 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1439022 T6534 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1439023 T6534 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1439032 T6534 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 1439033 T6534 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1439038 T6534 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1439039 T6534 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1439043 T6534 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1439044 T6534 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1439048 T6534 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1439049 T6534 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1439052 T6534 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1439053 T6534 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1439057 T6534 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1439058 T6534 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1439062 T6534 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1439063 T6534 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1439067 T6534 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1439068 T6534 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1439072 T6534 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1439072 T6534 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1439077 T6534 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1439078 T6534 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1439278 T6534 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1439281 T6534 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40903
   [junit4]   2> 1439282 T6534 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1439282 T6534 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1439282 T6534 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.TestRequestStatusCollectionAPI-controljetty-1395241343824
   [junit4]   2> 1439283 T6534 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.TestRequestStatusCollectionAPI-controljetty-1395241343824/'
   [junit4]   2> 1439293 T6534 oass.SolrDispatchFilter.loadConfigSolr Trying to read solr.xml from 127.0.0.1:43654/solr
   [junit4]   2> 1439294 T6534 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1439296 T6554 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@48d9b50c name:ZooKeeperConnection Watcher:127.0.0.1:43654/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1439296 T6534 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1439297 T6534 oass.SolrDispatchFilter.init ERROR Could not start Solr. Check solr/home property and the logs
   [junit4]   2> 1439299 T6534 oasc.SolrException.log ERROR null:org.apache.solr.common.SolrException: Could not load solr.xml from zookeeper
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.loadConfigSolr(SolrDispatchFilter.java:182)
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:198)
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
   [junit4]   2> 		at org.eclipse.jetty.servlet.FilterHolder.doStart(FilterHolder.java:119)
   [junit4]   2> 		at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:719)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1309)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1345)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1085)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:931)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:888)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:340)
   [junit4]   2> 		at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:325)
   [junit4]   2> 		at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:174)
   [junit4]   2> 		at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:65)
   [junit4]   2> 		at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:430)
   [junit4]   2> 		at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:403)
   [junit4]   2> 		at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJetty(AbstractFullDistribZkTestBase.java:490)
   [junit4]   2> 		at org.apache.solr.BaseDistributedSearchTestCase.createJetty(BaseDistributedSearchTestCase.java:356)
   [junit4]   2> 		at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:291)
   [junit4]   2> 		at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:868)
   [junit4]   2> 		at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
   [junit4]   2> 		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 		at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:826)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:862)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
   [junit4]   2> 		at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 		at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
   [junit4]   2> 		at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	Caused by: org.apache.solr.common.SolrException: Could not load solr.xml from zookeeper: node not found
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.loadConfigSolr(SolrDispatchFilter.java:178)
   [junit4]   2> 		... 58 more
   [junit4]   2> 	
   [junit4]   2> 1439300 T6534 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1439302 T6534 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1439303 T6556 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@24176450 name:ZooKeeperConnection Watcher:127.0.0.1:43654/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1439304 T6534 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1439304 T6534 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1439306 T6534 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1439308 T6534 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1439310 T6534 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 40903
   [junit4]   2> 1439331 T6534 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/_iev/da,null}
   [junit4]   2> 1439385 T6534 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 1439387 T6534 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:43654 43654
   [junit4]   2> 1439627 T6487 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1439722 T6471 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestRequestStatusCollectionAPI -Dtests.method=testDistribSearch -Dtests.seed=A01EA8B7EDDF9883 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_VE -Dtests.timezone=Etc/GMT+3 -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.84s J0 | TestRequestStatusCollectionAPI.testDistribSearch <<<
   [junit4]    > Throwable #1: org.apache.solr.common.cloud.ZooKeeperException: 
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A01EA8B7EDDF9883:21F826AF9A80F8BF]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.connect(CloudSolrServer.java:249)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.initCloud(AbstractFullDistribZkTestBase.java:256)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:312)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:868)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]    > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /live_nodes
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]    > 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:259)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:256)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:299)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.connect(CloudSolrServer.java:242)
   [junit4]    > 	... 41 more
   [junit4]   2> 1439726 T6534 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 1439726 T6534 oas.SolrTestCaseJ4.endTrackingZkClients ERROR ERROR: SolrZkClient opens=4 closes=3
   [junit4]   2> 845 T6533 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 3 leaked thread(s).
   [junit4]   2> 1439810 T6455 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1439810 T6556 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@24176450 name:ZooKeeperConnection Watcher:127.0.0.1:43654/solr got event WatchedEvent state:Disconnected type:None path:null path:null type:None
   [junit4]   2> 1439811 T6556 oascc.ConnectionManager.process zkClient has disconnected
   [junit4]   2> 1439904 T6439 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1439904 T6439 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1439904 T6439 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/testSolrCloudCollection/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:259)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 1439904 T6439 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1439905 T6439 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1439905 T6439 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1439905 T6439 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1439905 T6439 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1441656 T6555 oaz.ClientCnxn$SendThread.run WARN Session 0x144dadd5a940003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1443638 T6555 oaz.ClientCnxn$SendThread.run WARN Session 0x144dadd5a940003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1445081 T6555 oaz.ClientCnxn$SendThread.run WARN Session 0x144dadd5a940003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1446876 T6555 oaz.ClientCnxn$SendThread.run WARN Session 0x144dadd5a940003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1448072 T6555 oaz.ClientCnxn$SendThread.run WARN Session 0x144dadd5a940003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1449457 T6555 oaz.ClientCnxn$SendThread.run WARN Session 0x144dadd5a940003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1450752 T6555 oaz.ClientCnxn$SendThread.run WARN Session 0x144dadd5a940003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1451861 T6555 oaz.ClientCnxn$SendThread.run WARN Session 0x144dadd5a940003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1453457 T6555 oaz.ClientCnxn$SendThread.run WARN Session 0x144dadd5a940003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1454817 T6555 oaz.ClientCnxn$SendThread.run WARN Session 0x144dadd5a940003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1456338 T6555 oaz.ClientCnxn$SendThread.run WARN Session 0x144dadd5a940003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1457830 T6555 oaz.ClientCnxn$SendThread.run WARN Session 0x144dadd5a940003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1459415 T6555 oaz.ClientCnxn$SendThread.run WARN Session 0x144dadd5a940003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 21003 T6533 ccr.ThreadLeakControl.checkThreadLeaks SEVERE 2 threads leaked from SUITE scope at org.apache.solr.cloud.TestRequestStatusCollectionAPI: 
   [junit4]   2> 	   1) Thread[id=6555, name=TEST-TestRequestStatusCollectionAPI.testDistribSearch-seed#[A01EA8B7EDDF9883]-SendThread(localhost.localdomain:43654), state=TIMED_WAITING, group=TGRP-TestRequestStatusCollectionAPI]
   [junit4]   2> 	        at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 	   2) Thread[id=6556, name=TEST-TestRequestStatusCollectionAPI.testDistribSearch-seed#[A01EA8B7EDDF9883]-EventThread, state=WAITING, group=TGRP-TestRequestStatusCollectionAPI]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]   2> 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
   [junit4]   2> 21003 T6533 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads:
   [junit4]   2> 	   1) Thread[id=6555, name=TEST-TestRequestStatusCollectionAPI.testDistribSearch-seed#[A01EA8B7EDDF9883]-SendThread(localhost.localdomain:43654), state=TIMED_WAITING, group=TGRP-TestRequestStatusCollectionAPI]
   [junit4]   2> 	   2) Thread[id=6556, name=TEST-TestRequestStatusCollectionAPI.testDistribSearch-seed#[A01EA8B7EDDF9883]-EventThread, state=WAITING, group=TGRP-TestRequestStatusCollectionAPI]
   [junit4]   2> 1459891 T6555 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 1459891 T6556 oaz.ClientCnxn$EventThread.run ERROR Event thread exiting due to interruption java.lang.InterruptedException
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052)
   [junit4]   2> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
   [junit4]   2> 
   [junit4]   2> 1459892 T6555 oaz.ClientCnxn$SendThread.run WARN Session 0x144dadd5a940003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1460391 T6555 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 1460392 T6555 oaz.ClientCnxn$SendThread.run WARN Session 0x144dadd5a940003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1460891 T6555 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 1460893 T6555 oaz.ClientCnxn$SendThread.run WARN Session 0x144dadd5a940003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1461392 T6555 oaz.ClientCnxn$SendThread.run WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   [junit4]   2> 
   [junit4]   2> 1461892 T6555 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 1461893 T6555 oaz.ClientCnxn$SendThread.run WARN Session 0x144dadd5a940003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1462392 T6555 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 1462393 T6555 oaz.ClientCnxn$SendThread.run WARN Session 0x144dadd5a940003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 24007 T6533 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
   [junit4]   2> 	   1) Thread[id=6555, name=TEST-TestRequestStatusCollectionAPI.testDistribSearch-seed#[A01EA8B7EDDF9883]-SendThread(localhost.localdomain:43654), state=TIMED_WAITING, group=TGRP-TestRequestStatusCollectionAPI]
   [junit4]   2> 	        at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   [junit4]   2> NOTE: test params are: codec=Lucene45, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=es_VE, timezone=Etc/GMT+3
   [junit4]   2> NOTE: Linux 3.8.0-37-generic amd64/Oracle Corporation 1.7.0_51 (64-bit)/cpus=8,threads=2,free=152828800,total=295579648
   [junit4]   2> NOTE: All tests run in this JVM: [UpdateRequestProcessorFactoryTest, HdfsBasicDistributedZk2Test, AbstractAnalyticsStatsTest, TestSolrQueryParser, TestDynamicFieldCollectionResource, ShowFileRequestHandlerTest, TestNoOpRegenerator, DistributedQueryComponentCustomSortTest, TestLuceneMatchVersion, DeleteInactiveReplicaTest, TestGroupingSearch, TestInitQParser, DocValuesTest, TestSolrXMLSerializer, DistributedExpandComponentTest, TestPerFieldSimilarity, TestRandomDVFaceting, DirectUpdateHandlerOptimizeTest, SpatialFilterTest, LoggingHandlerTest, TestLFUCache, TestHighlightDedupGrouping, TestFreeTextSuggestions, TestPseudoReturnFields, TestDocSet, TestSearchPerf, RemoteQueryErrorTest, TriLevelCompositeIdRoutingTest, TestDefaultSearchFieldResource, AnalysisAfterCoreReloadTest, CachingDirectoryFactoryTest, TestRealTimeGet, OverseerTest, LegacyHTMLStripCharFilterTest, TestSort, TestLMJelinekMercerSimilarityFactory, TestFieldSortValues, ParsingFieldUpdateProcessorsTest, ZkCLITest, SearchHandlerTest, LeaderElectionTest, BlockCacheTest, SuggesterFSTTest, TestUniqueKeyFieldResource, StatelessScriptUpdateProcessorFactoryTest, TestSolrIndexConfig, TestExceedMaxTermLength, TestValueSourceCache, TermVectorComponentTest, SpellPossibilityIteratorTest, SchemaVersionSpecificBehaviorTest, TestDistributedMissingSort, MigrateRouteKeyTest, SolrInfoMBeanTest, QueryEqualityTest, TermsComponentTest, SyncSliceTest, CursorMarkTest, TestReloadAndDeleteDocs, CoreAdminHandlerTest, BinaryUpdateRequestHandlerTest, SOLR749Test, ResourceLoaderTest, QueryElevationComponentTest, SliceStateUpdateTest, ModifyConfFileTest, TestConfig, TestWriterPerf, TestCollapseQParserPlugin, TestManagedResourceStorage, UUIDFieldTest, BasicFunctionalityTest, TestClassNameShortening, TestAnalyzeInfixSuggestions, TestFunctionQuery, TestSurroundQueryParser, TestDocumentBuilder, TestJoin, HdfsBasicDistributedZkTest, TestJmxMonitoredMap, PeerSyncTest, TestSolrXml, TestPostingsSolrHighlighter, TestSystemIdResolver, TestAddFieldRealTimeGet, SuggesterTest, FunctionTest, SliceStateTest, TestBlendedInfixSuggestions, CacheHeaderTest, BasicDistributedZkTest, ClusterStateUpdateTest, DistributedSuggestComponentTest, DistributedQueryElevationComponentTest, UnloadDistributedZkTest, BasicZkTest, TestReplicationHandler, ZkSolrClientTest, ShardRoutingCustomTest, TestDistributedSearch, TestFaceting, TestHashPartitioner, TermVectorComponentDistributedTest, TestReload, HardAutoCommitTest, DistributedTermsComponentTest, TestCoreContainer, SolrCoreTest, SolrCmdDistributorTest, BadIndexSchemaTest, TestBadConfig, SoftAutoCommitTest, TestCSVLoader, WordBreakSolrSpellCheckerTest, FieldMutatingUpdateProcessorTest, DefaultValueUpdateProcessorTest, TestRemoteStreaming, XsltUpdateRequestHandlerTest, LukeRequestHandlerTest, DocumentAnalysisRequestHandlerTest, PrimitiveFieldTypeTest, DocumentBuilderTest, MoreLikeThisHandlerTest, IndexSchemaRuntimeFieldTest, IndexSchemaTest, JSONWriterTest, QueryParsingTest, TestPartialUpdateDeduplication, CSVRequestHandlerTest, UpdateParamsTest, CopyFieldTest, SolrIndexConfigTest, TestMergePolicyConfig, TestElisionMultitermQuery, TestSolrCoreProperties, NotRequiredUniqueKeyTest, TestCharFilters, SynonymTokenizerTest, TestNumberUtils, TestBM25SimilarityFactory, TestFastOutputStream, OpenExchangeRatesOrgProviderTest, PluginInfoTest, PrimUtilsTest, DOMUtilTest, ClusterStateTest, TestLRUCache, ZkNodePropsTest, TestRTGBase, SolrTestCaseJ4Test, NoFacetTest, ExpressionTest, AbstractAnalyticsFacetTest, FieldFacetExtrasTest, RangeFacetTest, AliasIntegrationTest, CollectionsAPIAsyncDistributedZkTest, CustomCollectionTest, OverseerRolesTest, TestDistribDocBasedVersion, TestRequestStatusCollectionAPI]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestRequestStatusCollectionAPI -Dtests.seed=A01EA8B7EDDF9883 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_VE -Dtests.timezone=Etc/GMT+3 -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J0 | TestRequestStatusCollectionAPI (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ERROR: SolrZkClient opens=4 closes=3
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A01EA8B7EDDF9883]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:435)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:180)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.TestRequestStatusCollectionAPI: 
   [junit4]    >    1) Thread[id=6555, name=TEST-TestRequestStatusCollectionAPI.testDistribSearch-seed#[A01EA8B7EDDF9883]-SendThread(localhost.localdomain:43654), state=TIMED_WAITING, group=TGRP-TestRequestStatusCollectionAPI]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]    >    2) Thread[id=6556, name=TEST-TestRequestStatusCollectionAPI.testDistribSearch-seed#[A01EA8B7EDDF9883]-EventThread, state=WAITING, group=TGRP-TestRequestStatusCollectionAPI]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A01EA8B7EDDF9883]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=6555, name=TEST-TestRequestStatusCollectionAPI.testDistribSearch-seed#[A01EA8B7EDDF9883]-SendThread(localhost.localdomain:43654), state=TIMED_WAITING, group=TGRP-TestRequestStatusCollectionAPI]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A01EA8B7EDDF9883]:0)
   [junit4] Completed on J0 in 24.07s, 1 test, 1 failure, 3 errors <<< FAILURES!

[...truncated 163 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1275: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 391 suites, 1645 tests, 3 suite-level errors, 1 error, 186 ignored (28 assumptions)

Total time: 55 minutes 24 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_51 -XX:+UseCompressedOops -XX:+UseSerialGC -XX:-UseSuperWord
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_51) - Build # 9846 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9846/
Java: 32bit/jdk1.7.0_51 -server -XX:+UseG1GC

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TriLevelCompositeIdRoutingTest

Error Message:
ERROR: SolrIndexSearcher opens=0 closes=1

Stack Trace:
java.lang.AssertionError: ERROR: SolrIndexSearcher opens=0 closes=1
	at __randomizedtesting.SeedInfo.seed([B4690ABD13609DAA]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.SolrTestCaseJ4.endTrackingSearchers(SolrTestCaseJ4.java:420)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:179)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:789)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TriLevelCompositeIdRoutingTest

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.TriLevelCompositeIdRoutingTest:     1) Thread[id=6127, name=TEST-TriLevelCompositeIdRoutingTest.testDistribSearch-seed#[B4690ABD13609DAA]-EventThread, state=WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)    2) Thread[id=6126, name=TEST-TriLevelCompositeIdRoutingTest.testDistribSearch-seed#[B4690ABD13609DAA]-SendThread(localhost.localdomain:46767), state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.TriLevelCompositeIdRoutingTest: 
   1) Thread[id=6127, name=TEST-TriLevelCompositeIdRoutingTest.testDistribSearch-seed#[B4690ABD13609DAA]-EventThread, state=WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
   2) Thread[id=6126, name=TEST-TriLevelCompositeIdRoutingTest.testDistribSearch-seed#[B4690ABD13609DAA]-SendThread(localhost.localdomain:46767), state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
	at __randomizedtesting.SeedInfo.seed([B4690ABD13609DAA]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TriLevelCompositeIdRoutingTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=6126, name=TEST-TriLevelCompositeIdRoutingTest.testDistribSearch-seed#[B4690ABD13609DAA]-SendThread(localhost.localdomain:46767), state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=6126, name=TEST-TriLevelCompositeIdRoutingTest.testDistribSearch-seed#[B4690ABD13609DAA]-SendThread(localhost.localdomain:46767), state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
	at __randomizedtesting.SeedInfo.seed([B4690ABD13609DAA]:0)


FAILED:  org.apache.solr.cloud.TriLevelCompositeIdRoutingTest.testDistribSearch

Error Message:


Stack Trace:
org.apache.solr.common.cloud.ZooKeeperException: 
	at __randomizedtesting.SeedInfo.seed([B4690ABD13609DAA:358F84A5643FFD96]:0)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.connect(CloudSolrServer.java:249)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.initCloud(AbstractFullDistribZkTestBase.java:256)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:312)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:868)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:826)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:862)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /live_nodes
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:259)
	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:256)
	at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:299)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.connect(CloudSolrServer.java:242)
	... 42 more




Build Log:
[...truncated 11285 lines...]
   [junit4] Suite: org.apache.solr.cloud.TriLevelCompositeIdRoutingTest
   [junit4]   2> 1716137 T6103 oas.SolrTestCaseJ4.startTrackingSearchers WARN startTrackingSearchers: numOpens=5 numCloses=4
   [junit4]   2> 1716137 T6103 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 1716137 T6103 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1716141 T6103 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TriLevelCompositeIdRoutingTest-1395234816643
   [junit4]   2> 1716142 T6103 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1716142 T6104 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1716243 T6103 oasc.ZkTestServer.run start zk server on port:46767
   [junit4]   2> 1716244 T6103 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1716246 T6110 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d40de1 name:ZooKeeperConnection Watcher:127.0.0.1:46767 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1716246 T6103 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1716246 T6103 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1716249 T6103 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1716250 T6112 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9b7ad name:ZooKeeperConnection Watcher:127.0.0.1:46767/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1716250 T6103 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1716251 T6103 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1716252 T6103 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1716254 T6103 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1716255 T6103 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1716257 T6103 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1716257 T6103 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1716260 T6103 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 1716261 T6103 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1716264 T6103 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1716264 T6103 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1716266 T6103 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1716266 T6103 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1716268 T6103 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1716269 T6103 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1716270 T6103 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1716271 T6103 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1716274 T6103 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1716274 T6103 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1716278 T6103 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1716278 T6103 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1716280 T6103 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1716281 T6103 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1716282 T6103 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1716283 T6103 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1716285 T6103 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1716285 T6103 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1716288 T6103 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1716289 T6114 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@78fbf2 name:ZooKeeperConnection Watcher:127.0.0.1:46767/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1716289 T6103 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1716491 T6103 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1716494 T6103 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1716495 T6103 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:59990
   [junit4]   2> 1716497 T6103 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1716497 T6103 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1716497 T6103 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.TriLevelCompositeIdRoutingTest-controljetty-1395234816792
   [junit4]   2> 1716498 T6103 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.TriLevelCompositeIdRoutingTest-controljetty-1395234816792/'
   [junit4]   2> 1716508 T6103 oass.SolrDispatchFilter.loadConfigSolr Trying to read solr.xml from 127.0.0.1:46767/solr
   [junit4]   2> 1716509 T6103 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1716510 T6125 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3811a3 name:ZooKeeperConnection Watcher:127.0.0.1:46767/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1716510 T6103 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1716511 T6103 oass.SolrDispatchFilter.init ERROR Could not start Solr. Check solr/home property and the logs
   [junit4]   2> 1716512 T6103 oasc.SolrException.log ERROR null:org.apache.solr.common.SolrException: Could not load solr.xml from zookeeper
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.loadConfigSolr(SolrDispatchFilter.java:182)
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:198)
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
   [junit4]   2> 		at org.eclipse.jetty.servlet.FilterHolder.doStart(FilterHolder.java:119)
   [junit4]   2> 		at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:719)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1309)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1345)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1085)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:931)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:888)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:340)
   [junit4]   2> 		at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:325)
   [junit4]   2> 		at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:174)
   [junit4]   2> 		at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:65)
   [junit4]   2> 		at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:430)
   [junit4]   2> 		at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:403)
   [junit4]   2> 		at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJetty(AbstractFullDistribZkTestBase.java:490)
   [junit4]   2> 		at org.apache.solr.BaseDistributedSearchTestCase.createJetty(BaseDistributedSearchTestCase.java:356)
   [junit4]   2> 		at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:291)
   [junit4]   2> 		at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:868)
   [junit4]   2> 		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 		at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   2> 		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 		at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:826)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:862)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
   [junit4]   2> 		at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 		at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
   [junit4]   2> 		at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	Caused by: org.apache.solr.common.SolrException: Could not load solr.xml from zookeeper: node not found
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.loadConfigSolr(SolrDispatchFilter.java:178)
   [junit4]   2> 		... 59 more
   [junit4]   2> 	
   [junit4]   2> 1716513 T6103 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1716515 T6103 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1716516 T6127 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1282c76 name:ZooKeeperConnection Watcher:127.0.0.1:46767/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1716517 T6103 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1716517 T6103 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1716519 T6103 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1716520 T6103 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1716521 T6103 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 59990
   [junit4]   2> 1716542 T6103 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 1716595 T6103 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 1716597 T6103 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:46767 46767
   [junit4]   2> 1717106 T6024 oasc.Overseer.createOverseerNode ERROR Could not create Overseer node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:311)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:308)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:308)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:1175)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer.getInQueue(Overseer.java:1139)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:191)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:422)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:422)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 1717106 T6024 oasc.SolrCore.close [testSolrCloudCollection_shard1_replica1]  CLOSING SolrCore org.apache.solr.core.SolrCore@922176
   [junit4]   2> 1717112 T6024 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
   [junit4]   2> 1717113 T6024 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 1717113 T6024 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1717113 T6024 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 1717114 T6024 oasc.SolrCore.closeSearcher [testSolrCloudCollection_shard1_replica1] Closing main searcher on request.
   [junit4]   2> 1717114 T6024 oasc.CachingDirectoryFactory.close Closing RAMDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1717114 T6024 oasc.CachingDirectoryFactory.closeCacheValue looking to close data/index [CachedDir<<refCount=0;path=data/index;done=false>>]
   [junit4]   2> 1717114 T6024 oasc.CachingDirectoryFactory.close Closing directory: data/index
   [junit4]   2> 1717114 T6024 oasc.CachingDirectoryFactory.closeCacheValue looking to close data [CachedDir<<refCount=0;path=data;done=false>>]
   [junit4]   2> 1717115 T6024 oasc.CachingDirectoryFactory.close Closing directory: data
   [junit4]   2> 1717115 T6024 oasc.LeaderElector$1.process WARN  java.lang.RuntimeException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer
   [junit4]   2> 	at org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:1184)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer.getInQueue(Overseer.java:1139)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:191)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:422)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:422)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:311)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:308)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:308)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:1175)
   [junit4]   2> 	... 18 more
   [junit4]   2> 
   [junit4]   2> 1717115 T6024 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1717116 T6024 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1717116 T6024 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1717215 T6008 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1717216 T6008 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1717216 T6008 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1717216 T6008 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1726692 T6105 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x144da79c2850004, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TriLevelCompositeIdRoutingTest -Dtests.method=testDistribSearch -Dtests.seed=B4690ABD13609DAA -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_TN -Dtests.timezone=Asia/Kashgar -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   10.6s J0 | TriLevelCompositeIdRoutingTest.testDistribSearch <<<
   [junit4]    > Throwable #1: org.apache.solr.common.cloud.ZooKeeperException: 
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([B4690ABD13609DAA:358F84A5643FFD96]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.connect(CloudSolrServer.java:249)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.initCloud(AbstractFullDistribZkTestBase.java:256)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:312)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:868)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]    > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /live_nodes
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]    > 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:259)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:256)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:299)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.connect(CloudSolrServer.java:242)
   [junit4]    > 	... 42 more
   [junit4]   2> 1726711 T6103 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 1726792 T6127 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1282c76 name:ZooKeeperConnection Watcher:127.0.0.1:46767/solr got event WatchedEvent state:Disconnected type:None path:null path:null type:None
   [junit4]   2> 1726792 T6127 oascc.ConnectionManager.process zkClient has disconnected
   [junit4]   2> 1728019 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1729904 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1731218 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1733020 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1734349 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1735706 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1736832 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1737968 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1739211 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1741098 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1742919 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1744041 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1745678 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1746935 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1748384 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1750417 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1752431 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1754522 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1756021 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1757359 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1758559 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1759695 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1761617 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1763407 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1765208 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1767298 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1768460 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1770395 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1772171 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1773933 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1775770 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1776998 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1778886 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1780396 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1782374 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1784200 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1785341 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1786714 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1788790 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1790701 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1792149 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1793337 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1794869 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1796309 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1798127 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1799977 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1801924 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1803333 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1805189 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1806536 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1808068 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1810000 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1811175 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1812367 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1813803 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1815270 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1816469 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1817656 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1819709 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1820949 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1822963 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1824076 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1825609 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1827211 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1828797 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1829971 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1831620 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1833109 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1834570 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1836335 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1838343 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1840352 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1842181 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1843335 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1844453 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1845979 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1847725 T6103 oas.SolrTestCaseJ4.endTrackingSearchers ERROR ERROR: SolrIndexSearcher opens=0 closes=1
   [junit4]   2> 131592 T6102 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> 1847945 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1849112 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1850541 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1852170 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1854016 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1855277 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1856544 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$Se

[...truncated too long message...]

unit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1870319 T6126 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 1870320 T6126 oaz.ClientCnxn$SendThread.run WARN Session 0x144da79c2850004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 154683 T6102 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
   [junit4]   2> 	   1) Thread[id=6126, name=TEST-TriLevelCompositeIdRoutingTest.testDistribSearch-seed#[B4690ABD13609DAA]-SendThread(localhost.localdomain:46767), state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
   [junit4]   2> 	        at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   [junit4]   2> NOTE: test params are: codec=HighCompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=HIGH_COMPRESSION, chunkSize=149), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=HIGH_COMPRESSION, chunkSize=149)), sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=ar_TN, timezone=Asia/Kashgar
   [junit4]   2> NOTE: Linux 3.8.0-37-generic i386/Oracle Corporation 1.7.0_51 (32-bit)/cpus=8,threads=2,free=391125864,total=535822336
   [junit4]   2> NOTE: All tests run in this JVM: [TermsComponentTest, TestSort, FieldMutatingUpdateProcessorTest, BasicDistributedZkTest, DOMUtilTest, QueryEqualityTest, WordBreakSolrSpellCheckerTest, TestSurroundQueryParser, TestRealTimeGet, TestManagedSchemaFieldResource, TestRandomDVFaceting, TestCoreContainer, MigrateRouteKeyTest, TestSolrXMLSerializer, CacheHeaderTest, ClusterStateUpdateTest, StatelessScriptUpdateProcessorFactoryTest, ModifyConfFileTest, OverseerCollectionProcessorTest, HdfsBasicDistributedZkTest, RemoteQueryErrorTest, SampleTest, ScriptEngineTest, LegacyHTMLStripCharFilterTest, SearchHandlerTest, FileBasedSpellCheckerTest, SpatialFilterTest, TestLRUCache, TestAddFieldRealTimeGet, CoreAdminRequestStatusTest, CoreMergeIndexesAdminHandlerTest, DistributedQueryComponentCustomSortTest, TestLFUCache, DistributedQueryElevationComponentTest, TestUniqueKeyFieldResource, EchoParamsTest, CSVRequestHandlerTest, TestNoOpRegenerator, TestNonNRTOpen, TestQueryUtils, SuggesterFSTTest, TestJmxMonitoredMap, SimpleFacetsTest, SpellPossibilityIteratorTest, BasicFunctionalityTest, FunctionTest, TestFunctionQuery, StressHdfsTest, SystemInfoHandlerTest, SolrInfoMBeanTest, TestCollationFieldDocValues, TestAnalyzeInfixSuggestions, TestBadConfig, TestElisionMultitermQuery, ShardSplitTest, ZkControllerTest, BadComponentTest, TestReloadAndDeleteDocs, CachingDirectoryFactoryTest, ClusterStateTest, LoggingHandlerTest, SolrIndexSplitterTest, UpdateParamsTest, ShowFileRequestHandlerTest, TestSuggestSpellingConverter, EnumFieldTest, TestImplicitCoreProperties, DistributedExpandComponentTest, TestPerFieldSimilarity, TestRemoteStreaming, AddSchemaFieldsUpdateProcessorFactoryTest, SchemaVersionSpecificBehaviorTest, TestSystemIdResolver, TestGroupingSearch, DocumentBuilderTest, ResourceLoaderTest, BinaryUpdateRequestHandlerTest, TestSchemaSimilarityResource, TestSolrIndexConfig, RecoveryZkTest, ExternalFileFieldSortTest, TestLMDirichletSimilarityFactory, TestReplicationHandler, CopyFieldTest, CoreAdminHandlerTest, TestStandardQParsers, URLClassifyProcessorTest, ResponseLogComponentTest, DistributedQueryComponentOptimizationTest, TestDocumentBuilder, AsyncMigrateRouteKeyTest, TestDynamicFieldCollectionResource, PeerSyncTest, LeaderElectionTest, SyncSliceTest, ShardRoutingTest, FullSolrCloudDistribCmdsTest, TestZkChroot, SliceStateUpdateTest, AutoCommitTest, TestDistributedGrouping, DistributedSpellCheckComponentTest, TermVectorComponentDistributedTest, TestMultiCoreConfBootstrap, TestStressVersions, DistributedTermsComponentTest, SolrCoreTest, SolrCmdDistributorTest, BadIndexSchemaTest, TestLazyCores, AnalysisAfterCoreReloadTest, SignatureUpdateProcessorFactoryTest, TestExtendedDismaxParser, SpellCheckCollatorTest, TestFoldingMultitermQuery, SuggesterTest, TestTrie, SuggesterWFSTTest, SolrCoreCheckLockOnStartupTest, TestUpdate, SortByFunctionTest, XsltUpdateRequestHandlerTest, LukeRequestHandlerTest, TestReversedWildcardFilterFactory, TestQueryTypes, TestOmitPositions, PathHierarchyTokenizerFactoryTest, TestIndexingPerformance, FieldAnalysisRequestHandlerTest, FastVectorHighlighterTest, SolrPluginUtilsTest, TestJmxIntegration, UpdateRequestProcessorFactoryTest, MBeansHandlerTest, TestAnalyzedSuggestions, TestPHPSerializedResponseWriter, TestComponentsName, SOLR749Test, SolrIndexConfigTest, TestSolrDeletionPolicy2, TestBinaryField, TestConfig, TestSolrCoreProperties, DirectSolrConnectionTest, TestLuceneMatchVersion, TestCodecSupport, TestXIncludeConfig, TestBM25SimilarityFactory, DateMathParserTest, ChaosMonkeyNothingIsSafeTest, TestDistribDocBasedVersion, TriLevelCompositeIdRoutingTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TriLevelCompositeIdRoutingTest -Dtests.seed=B4690ABD13609DAA -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_TN -Dtests.timezone=Asia/Kashgar -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J0 | TriLevelCompositeIdRoutingTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ERROR: SolrIndexSearcher opens=0 closes=1
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([B4690ABD13609DAA]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.endTrackingSearchers(SolrTestCaseJ4.java:420)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:179)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.TriLevelCompositeIdRoutingTest: 
   [junit4]    >    1) Thread[id=6127, name=TEST-TriLevelCompositeIdRoutingTest.testDistribSearch-seed#[B4690ABD13609DAA]-EventThread, state=WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
   [junit4]    >    2) Thread[id=6126, name=TEST-TriLevelCompositeIdRoutingTest.testDistribSearch-seed#[B4690ABD13609DAA]-SendThread(localhost.localdomain:46767), state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([B4690ABD13609DAA]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=6126, name=TEST-TriLevelCompositeIdRoutingTest.testDistribSearch-seed#[B4690ABD13609DAA]-SendThread(localhost.localdomain:46767), state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([B4690ABD13609DAA]:0)
   [junit4] Completed on J0 in 154.72s, 1 test, 1 failure, 3 errors <<< FAILURES!

[...truncated 11 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1275: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 391 suites, 1645 tests, 3 suite-level errors, 1 error, 49 ignored (28 assumptions)

Total time: 57 minutes 53 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_51 -server -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0) - Build # 9845 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9845/
Java: 32bit/jdk1.8.0 -client -XX:+UseConcMarkSweepGC

33 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.SolrTestCaseJ4Test

Error Message:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms

Stack Trace:
java.lang.RuntimeException: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at __randomizedtesting.SeedInfo.seed([E368A8EA0D0F1923]:0)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:142)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:147)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:98)
	at org.apache.solr.SolrTestCaseJ4.createCore(SolrTestCaseJ4.java:566)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:556)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:365)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:372)
	at org.apache.solr.SolrTestCaseJ4Test.beforeClass(SolrTestCaseJ4Test.java:51)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:766)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:96)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:83)
	at org.apache.solr.cloud.ZkController.checkChrootPath(ZkController.java:667)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:133)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:67)
	at org.apache.solr.core.CoreContainer.load(CoreContainer.java:230)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:137)
	... 29 more
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
	... 37 more


FAILED:  junit.framework.TestSuite.org.apache.solr.analysis.TestCharFilters

Error Message:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms

Stack Trace:
java.lang.RuntimeException: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at __randomizedtesting.SeedInfo.seed([E368A8EA0D0F1923]:0)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:142)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:147)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:98)
	at org.apache.solr.SolrTestCaseJ4.createCore(SolrTestCaseJ4.java:566)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:556)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:365)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:352)
	at org.apache.solr.analysis.TestCharFilters.beforeClass(TestCharFilters.java:31)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:766)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:96)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:83)
	at org.apache.solr.cloud.ZkController.checkChrootPath(ZkController.java:667)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:133)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:67)
	at org.apache.solr.core.CoreContainer.load(CoreContainer.java:230)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:137)
	... 29 more
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
	... 37 more


FAILED:  junit.framework.TestSuite.org.apache.solr.analytics.NoFacetTest

Error Message:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms

Stack Trace:
java.lang.RuntimeException: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at __randomizedtesting.SeedInfo.seed([E368A8EA0D0F1923]:0)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:142)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:147)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:98)
	at org.apache.solr.SolrTestCaseJ4.createCore(SolrTestCaseJ4.java:566)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:556)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:365)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:352)
	at org.apache.solr.analytics.NoFacetTest.beforeClass(NoFacetTest.java:66)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:766)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:96)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:83)
	at org.apache.solr.cloud.ZkController.checkChrootPath(ZkController.java:667)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:133)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:67)
	at org.apache.solr.core.CoreContainer.load(CoreContainer.java:230)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:137)
	... 29 more
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
	... 37 more


FAILED:  junit.framework.TestSuite.org.apache.solr.analytics.facet.FieldFacetExtrasTest

Error Message:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms

Stack Trace:
java.lang.RuntimeException: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at __randomizedtesting.SeedInfo.seed([E368A8EA0D0F1923]:0)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:142)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:147)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:98)
	at org.apache.solr.SolrTestCaseJ4.createCore(SolrTestCaseJ4.java:566)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:556)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:365)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:352)
	at org.apache.solr.analytics.facet.FieldFacetExtrasTest.beforeClass(FieldFacetExtrasTest.java:50)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:766)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:96)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:83)
	at org.apache.solr.cloud.ZkController.checkChrootPath(ZkController.java:667)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:133)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:67)
	at org.apache.solr.core.CoreContainer.load(CoreContainer.java:230)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:137)
	... 29 more
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
	... 37 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest

Error Message:
ERROR: SolrZkClient opens=4 closes=3

Stack Trace:
java.lang.AssertionError: ERROR: SolrZkClient opens=4 closes=3
	at __randomizedtesting.SeedInfo.seed([E368A8EA0D0F1923]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:435)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:180)
	at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:789)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest

Error Message:
15 threads leaked from SUITE scope at org.apache.solr.cloud.RecoveryZkTest:     1) Thread[id=1461, name=TEST-RecoveryZkTest.testDistribSearch-seed#[E368A8EA0D0F1923]-EventThread, state=WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)    2) Thread[id=1441, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:178)         at java.lang.Thread.run(Thread.java:744)    3) Thread[id=1449, name=HashSessionScavenger-42, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at java.lang.Object.wait(Native Method)         at java.util.TimerThread.mainLoop(Timer.java:552)         at java.util.TimerThread.run(Timer.java:505)    4) Thread[id=1451, name=qtp25170843-1451 Selector1, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)         at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:744)    5) Thread[id=1453, name=qtp25170843-1453 Acceptor1 SelectChannelConnector@127.0.0.1:51416, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:224)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:744)    6) Thread[id=1456, name=qtp25170843-1456, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)         at java.lang.Thread.run(Thread.java:744)    7) Thread[id=1457, name=qtp25170843-1457, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)         at java.lang.Thread.run(Thread.java:744)    8) Thread[id=1455, name=qtp25170843-1455, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)         at java.lang.Thread.run(Thread.java:744)    9) Thread[id=1442, name=SessionTracker, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at java.lang.Object.wait(Native Method)         at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)   10) Thread[id=1440, name=Thread-581, state=WAITING, group=TGRP-RecoveryZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Thread.join(Thread.java:1244)         at java.lang.Thread.join(Thread.java:1318)         at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:302)         at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:107)         at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:221)   11) Thread[id=1444, name=ProcessThread(sid:0 cport:-1):, state=WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:120)   12) Thread[id=1450, name=qtp25170843-1450 Selector0, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)         at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:744)   13) Thread[id=1460, name=TEST-RecoveryZkTest.testDistribSearch-seed#[E368A8EA0D0F1923]-SendThread(localhost.localdomain:40005), state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)   14) Thread[id=1452, name=qtp25170843-1452 Acceptor0 SelectChannelConnector@127.0.0.1:51416, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:744)   15) Thread[id=1454, name=qtp25170843-1454, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)         at java.lang.Thread.run(Thread.java:744)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 15 threads leaked from SUITE scope at org.apache.solr.cloud.RecoveryZkTest: 
   1) Thread[id=1461, name=TEST-RecoveryZkTest.testDistribSearch-seed#[E368A8EA0D0F1923]-EventThread, state=WAITING, group=TGRP-RecoveryZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
   2) Thread[id=1441, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:178)
        at java.lang.Thread.run(Thread.java:744)
   3) Thread[id=1449, name=HashSessionScavenger-42, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]
        at java.lang.Object.wait(Native Method)
        at java.util.TimerThread.mainLoop(Timer.java:552)
        at java.util.TimerThread.run(Timer.java:505)
   4) Thread[id=1451, name=qtp25170843-1451 Selector1, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
        at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:744)
   5) Thread[id=1453, name=qtp25170843-1453 Acceptor1 SelectChannelConnector@127.0.0.1:51416, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:224)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:744)
   6) Thread[id=1456, name=qtp25170843-1456, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
        at java.lang.Thread.run(Thread.java:744)
   7) Thread[id=1457, name=qtp25170843-1457, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
        at java.lang.Thread.run(Thread.java:744)
   8) Thread[id=1455, name=qtp25170843-1455, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
        at java.lang.Thread.run(Thread.java:744)
   9) Thread[id=1442, name=SessionTracker, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]
        at java.lang.Object.wait(Native Method)
        at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)
  10) Thread[id=1440, name=Thread-581, state=WAITING, group=TGRP-RecoveryZkTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Thread.join(Thread.java:1244)
        at java.lang.Thread.join(Thread.java:1318)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:302)
        at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:107)
        at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:221)
  11) Thread[id=1444, name=ProcessThread(sid:0 cport:-1):, state=WAITING, group=TGRP-RecoveryZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:120)
  12) Thread[id=1450, name=qtp25170843-1450 Selector0, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
        at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:744)
  13) Thread[id=1460, name=TEST-RecoveryZkTest.testDistribSearch-seed#[E368A8EA0D0F1923]-SendThread(localhost.localdomain:40005), state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
  14) Thread[id=1452, name=qtp25170843-1452 Acceptor0 SelectChannelConnector@127.0.0.1:51416, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:744)
  15) Thread[id=1454, name=qtp25170843-1454, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
        at java.lang.Thread.run(Thread.java:744)
	at __randomizedtesting.SeedInfo.seed([E368A8EA0D0F1923]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=1450, name=qtp25170843-1450 Selector0, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)         at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317)         at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:193)         at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213)         at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:78)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:105)         at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:536)         at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:744)    2) Thread[id=1460, name=TEST-RecoveryZkTest.testDistribSearch-seed#[E368A8EA0D0F1923]-SendThread(localhost.localdomain:40005), state=RUNNABLE, group=TGRP-RecoveryZkTest]         at java.security.AccessController.getStackAccessControlContext(Native Method)         at java.security.AccessController.checkPermission(AccessController.java:858)         at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)         at javax.security.auth.login.Configuration.getConfiguration(Configuration.java:228)         at org.apache.zookeeper.client.ZooKeeperSaslClient.clientTunneledAuthenticationInProgress(ZooKeeperSaslClient.java:487)         at org.apache.zookeeper.ClientCnxn$SendThread.clientTunneledAuthenticationInProgress(ClientCnxn.java:1241)         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:360)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)    3) Thread[id=1452, name=qtp25170843-1452 Acceptor0 SelectChannelConnector@127.0.0.1:51416, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at org.eclipse.jetty.server.nio.SelectChannelConnector.getConnection(SelectChannelConnector.java:160)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:934)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:744)    4) Thread[id=1441, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at java.lang.Thread.isInterrupted(Native Method)         at java.lang.Thread.isInterrupted(Thread.java:960)         at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:218)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:78)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:178)         at java.lang.Thread.run(Thread.java:744)    5) Thread[id=1449, name=HashSessionScavenger-42, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at java.lang.Object.wait(Native Method)         at java.util.TimerThread.mainLoop(Timer.java:552)         at java.util.TimerThread.run(Timer.java:505)    6) Thread[id=1451, name=qtp25170843-1451 Selector1, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)         at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317)         at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:193)         at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213)         at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:78)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)         at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:744)    7) Thread[id=1453, name=qtp25170843-1453 Acceptor1 SelectChannelConnector@127.0.0.1:51416, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:102)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:744)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=1450, name=qtp25170843-1450 Selector0, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
        at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317)
        at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:193)
        at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213)
        at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:78)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:105)
        at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:536)
        at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:744)
   2) Thread[id=1460, name=TEST-RecoveryZkTest.testDistribSearch-seed#[E368A8EA0D0F1923]-SendThread(localhost.localdomain:40005), state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at java.security.AccessController.getStackAccessControlContext(Native Method)
        at java.security.AccessController.checkPermission(AccessController.java:858)
        at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
        at javax.security.auth.login.Configuration.getConfiguration(Configuration.java:228)
        at org.apache.zookeeper.client.ZooKeeperSaslClient.clientTunneledAuthenticationInProgress(ZooKeeperSaslClient.java:487)
        at org.apache.zookeeper.ClientCnxn$SendThread.clientTunneledAuthenticationInProgress(ClientCnxn.java:1241)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:360)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   3) Thread[id=1452, name=qtp25170843-1452 Acceptor0 SelectChannelConnector@127.0.0.1:51416, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at org.eclipse.jetty.server.nio.SelectChannelConnector.getConnection(SelectChannelConnector.java:160)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:934)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:744)
   4) Thread[id=1441, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at java.lang.Thread.isInterrupted(Native Method)
        at java.lang.Thread.isInterrupted(Thread.java:960)
        at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:218)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:78)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:178)
        at java.lang.Thread.run(Thread.java:744)
   5) Thread[id=1449, name=HashSessionScavenger-42, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]
        at java.lang.Object.wait(Native Method)
        at java.util.TimerThread.mainLoop(Timer.java:552)
        at java.util.TimerThread.run(Timer.java:505)
   6) Thread[id=1451, name=qtp25170843-1451 Selector1, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
        at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317)
        at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:193)
        at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213)
        at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:78)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
        at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:744)
   7) Thread[id=1453, name=qtp25170843-1453 Acceptor1 SelectChannelConnector@127.0.0.1:51416, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:102)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:744)
	at __randomizedtesting.SeedInfo.seed([E368A8EA0D0F1923]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestConfig

Error Message:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms

Stack Trace:
java.lang.RuntimeException: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at __randomizedtesting.SeedInfo.seed([E368A8EA0D0F1923]:0)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:142)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:147)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:98)
	at org.apache.solr.SolrTestCaseJ4.createCore(SolrTestCaseJ4.java:566)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:556)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:365)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:352)
	at org.apache.solr.core.TestConfig.beforeClass(TestConfig.java:45)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:766)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:96)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:83)
	at org.apache.solr.cloud.ZkController.checkChrootPath(ZkController.java:667)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:133)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:67)
	at org.apache.solr.core.CoreContainer.load(CoreContainer.java:230)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:137)
	... 29 more
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
	... 37 more


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestCoreContainer

Error Message:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms

Stack Trace:
java.lang.RuntimeException: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at __randomizedtesting.SeedInfo.seed([E368A8EA0D0F1923]:0)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:142)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:147)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:98)
	at org.apache.solr.SolrTestCaseJ4.createCore(SolrTestCaseJ4.java:566)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:556)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:365)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:352)
	at org.apache.solr.core.TestCoreContainer.beforeClass(TestCoreContainer.java:54)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:766)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:96)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:83)
	at org.apache.solr.cloud.ZkController.checkChrootPath(ZkController.java:667)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:133)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:67)
	at org.apache.solr.core.CoreContainer.load(CoreContainer.java:230)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:137)
	... 29 more
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
	... 37 more


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.JsonLoaderTest

Error Message:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms

Stack Trace:
java.lang.RuntimeException: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at __randomizedtesting.SeedInfo.seed([E368A8EA0D0F1923]:0)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:142)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:147)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:98)
	at org.apache.solr.SolrTestCaseJ4.createCore(SolrTestCaseJ4.java:566)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:556)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:365)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:352)
	at org.apache.solr.handler.JsonLoaderTest.beforeTests(JsonLoaderTest.java:44)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:766)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:96)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:83)
	at org.apache.solr.cloud.ZkController.checkChrootPath(ZkController.java:667)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:133)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:67)
	at org.apache.solr.core.CoreContainer.load(CoreContainer.java:230)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:137)
	... 29 more
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
	... 37 more


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.admin.LoggingHandlerTest

Error Message:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms

Stack Trace:
java.lang.RuntimeException: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at __randomizedtesting.SeedInfo.seed([E368A8EA0D0F1923]:0)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:142)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:147)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:98)
	at org.apache.solr.SolrTestCaseJ4.createCore(SolrTestCaseJ4.java:566)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:556)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:365)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:352)
	at org.apache.solr.handler.admin.LoggingHandlerTest.beforeClass(LoggingHandlerTest.java:39)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:766)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:96)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:83)
	at org.apache.solr.cloud.ZkController.checkChrootPath(ZkController.java:667)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:133)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:67)
	at org.apache.solr.core.CoreContainer.load(CoreContainer.java:230)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:137)
	... 29 more
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
	... 37 more


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.component.SearchHandlerTest

Error Message:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms

Stack Trace:
java.lang.RuntimeException: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at __randomizedtesting.SeedInfo.seed([E368A8EA0D0F1923]:0)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:142)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:147)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:98)
	at org.apache.solr.SolrTestCaseJ4.createCore(SolrTestCaseJ4.java:566)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:556)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:365)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:352)
	at org.apache.solr.handler.component.SearchHandlerTest.beforeTests(SearchHandlerTest.java:34)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:766)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:96)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:83)
	at org.apache.solr.cloud.ZkController.checkChrootPath(ZkController.java:667)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:133)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:67)
	at org.apache.solr.core.CoreContainer.load(CoreContainer.java:230)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:137)
	... 29 more
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
	... 37 more


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.component.SuggestComponentTest

Error Message:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms

Stack Trace:
java.lang.RuntimeException: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at __randomizedtesting.SeedInfo.seed([E368A8EA0D0F1923]:0)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:142)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:147)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:98)
	at org.apache.solr.SolrTestCaseJ4.createCore(SolrTestCaseJ4.java:566)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:556)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:365)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:352)
	at org.apache.solr.handler.component.SuggestComponentTest.beforeClass(SuggestComponentTest.java:32)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:766)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:96)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:83)
	at org.apache.solr.cloud.ZkController.checkChrootPath(ZkController.java:667)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:133)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:67)
	at org.apache.solr.core.CoreContainer.load(CoreContainer.java:230)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:137)
	... 29 more
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
	... 37 more


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.component.TermsComponentTest

Error Message:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms

Stack Trace:
java.lang.RuntimeException: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at __randomizedtesting.SeedInfo.seed([E368A8EA0D0F1923]:0)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:142)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:147)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:98)
	at org.apache.solr.SolrTestCaseJ4.createCore(SolrTestCaseJ4.java:566)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:556)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:365)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:352)
	at org.apache.solr.handler.component.TermsComponentTest.beforeTest(TermsComponentTest.java:36)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:766)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:96)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:83)
	at org.apache.solr.cloud.ZkController.checkChrootPath(ZkController.java:667)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:133)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:67)
	at org.apache.solr.core.CoreContainer.load(CoreContainer.java:230)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:137)
	... 29 more
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
	... 37 more


FAILED:  junit.framework.TestSuite.org.apache.solr.request.TestWriterPerf

Error Message:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms

Stack Trace:
java.lang.RuntimeException: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at __randomizedtesting.SeedInfo.seed([E368A8EA0D0F1923]:0)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:142)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:147)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:98)
	at org.apache.solr.SolrTestCaseJ4.createCore(SolrTestCaseJ4.java:566)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:556)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:365)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:352)
	at org.apache.solr.request.TestWriterPerf.beforeClass(TestWriterPerf.java:45)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:766)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:96)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:83)
	at org.apache.solr.cloud.ZkController.checkChrootPath(ZkController.java:667)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:133)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:67)
	at org.apache.solr.core.CoreContainer.load(CoreContainer.java:230)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:137)
	... 29 more
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
	... 37 more


FAILED:  junit.framework.TestSuite.org.apache.solr.schema.DocValuesTest

Error Message:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms

Stack Trace:
java.lang.RuntimeException: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at __randomizedtesting.SeedInfo.seed([E368A8EA0D0F1923]:0)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:142)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:147)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:98)
	at org.apache.solr.SolrTestCaseJ4.createCore(SolrTestCaseJ4.java:566)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:556)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:365)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:352)
	at org.apache.solr.schema.DocValuesTest.beforeTests(DocValuesTest.java:36)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:766)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:96)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:83)
	at org.apache.solr.cloud.ZkController.checkChrootPath(ZkController.java:667)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:133)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:67)
	at org.apache.solr.core.CoreContainer.load(CoreContainer.java:230)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:137)
	... 29 more
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
	... 37 more


FAILED:  junit.framework.TestSuite.org.apache.solr.schema.SynonymTokenizerTest

Error Message:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms

Stack Trace:
java.lang.RuntimeException: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:36628 within 30000 ms
	at __randomizedtesting.SeedInfo.seed([E368A8EA0D0F1923]:0)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:142)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:147)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:98)
	at org.apache.solr.SolrTestCaseJ4.createCore(SolrTestCaseJ4.java:566)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:556)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:365)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:352)
	at org.apache.solr.schema.SynonymTokenizerTest.beforeTests(SynonymTokenizerTest.java:35)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:766)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapte

[...truncated too long message...]

at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]    >   14) Thread[id=1452, name=qtp25170843-1452 Acceptor0 SelectChannelConnector@127.0.0.1:51416, state=RUNNABLE, group=TGRP-RecoveryZkTest]
   [junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
   [junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
   [junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
   [junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >   15) Thread[id=1454, name=qtp25170843-1454, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   [junit4]    >         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E368A8EA0D0F1923]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=1450, name=qtp25170843-1450 Selector0, state=RUNNABLE, group=TGRP-RecoveryZkTest]
   [junit4]    >         at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
   [junit4]    >         at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317)
   [junit4]    >         at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:193)
   [junit4]    >         at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213)
   [junit4]    >         at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219)
   [junit4]    >         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:78)
   [junit4]    >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]    >         at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:105)
   [junit4]    >         at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:536)
   [junit4]    >         at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    2) Thread[id=1460, name=TEST-RecoveryZkTest.testDistribSearch-seed#[E368A8EA0D0F1923]-SendThread(localhost.localdomain:40005), state=RUNNABLE, group=TGRP-RecoveryZkTest]
   [junit4]    >         at java.security.AccessController.getStackAccessControlContext(Native Method)
   [junit4]    >         at java.security.AccessController.checkPermission(AccessController.java:858)
   [junit4]    >         at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    >         at javax.security.auth.login.Configuration.getConfiguration(Configuration.java:228)
   [junit4]    >         at org.apache.zookeeper.client.ZooKeeperSaslClient.clientTunneledAuthenticationInProgress(ZooKeeperSaslClient.java:487)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.clientTunneledAuthenticationInProgress(ClientCnxn.java:1241)
   [junit4]    >         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:360)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]    >    3) Thread[id=1452, name=qtp25170843-1452 Acceptor0 SelectChannelConnector@127.0.0.1:51416, state=RUNNABLE, group=TGRP-RecoveryZkTest]
   [junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.getConnection(SelectChannelConnector.java:160)
   [junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:934)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    4) Thread[id=1441, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-RecoveryZkTest]
   [junit4]    >         at java.lang.Thread.isInterrupted(Native Method)
   [junit4]    >         at java.lang.Thread.isInterrupted(Thread.java:960)
   [junit4]    >         at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:218)
   [junit4]    >         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:78)
   [junit4]    >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]    >         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:178)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    5) Thread[id=1449, name=HashSessionScavenger-42, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at java.util.TimerThread.mainLoop(Timer.java:552)
   [junit4]    >         at java.util.TimerThread.run(Timer.java:505)
   [junit4]    >    6) Thread[id=1451, name=qtp25170843-1451 Selector1, state=RUNNABLE, group=TGRP-RecoveryZkTest]
   [junit4]    >         at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
   [junit4]    >         at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317)
   [junit4]    >         at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:193)
   [junit4]    >         at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213)
   [junit4]    >         at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219)
   [junit4]    >         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:78)
   [junit4]    >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]    >         at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
   [junit4]    >         at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    7) Thread[id=1453, name=qtp25170843-1453 Acceptor1 SelectChannelConnector@127.0.0.1:51416, state=RUNNABLE, group=TGRP-RecoveryZkTest]
   [junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:102)
   [junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E368A8EA0D0F1923]:0)
   [junit4] Completed on J0 in 23.70s, 1 test, 1 failure, 3 errors <<< FAILURES!

[...truncated 637 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1275: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 391 suites, 1515 tests, 25 suite-level errors, 8 errors, 802 ignored (27 assumptions)

Total time: 60 minutes 3 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.8.0 -client -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0) - Build # 9844 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9844/
Java: 32bit/jdk1.8.0 -client -XX:+UseSerialGC

12 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.AliasIntegrationTest

Error Message:
ERROR: SolrZkClient opens=5 closes=4

Stack Trace:
java.lang.AssertionError: ERROR: SolrZkClient opens=5 closes=4
	at __randomizedtesting.SeedInfo.seed([7C4E9CA1B8B0EDAC]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:435)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:180)
	at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:789)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.AliasIntegrationTest

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.AliasIntegrationTest:     1) Thread[id=3003, name=TEST-AliasIntegrationTest.testDistribSearch-seed#[7C4E9CA1B8B0EDAC]-EventThread, state=WAITING, group=TGRP-AliasIntegrationTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)    2) Thread[id=3002, name=TEST-AliasIntegrationTest.testDistribSearch-seed#[7C4E9CA1B8B0EDAC]-SendThread(localhost.localdomain:36192), state=TIMED_WAITING, group=TGRP-AliasIntegrationTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.AliasIntegrationTest: 
   1) Thread[id=3003, name=TEST-AliasIntegrationTest.testDistribSearch-seed#[7C4E9CA1B8B0EDAC]-EventThread, state=WAITING, group=TGRP-AliasIntegrationTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
   2) Thread[id=3002, name=TEST-AliasIntegrationTest.testDistribSearch-seed#[7C4E9CA1B8B0EDAC]-SendThread(localhost.localdomain:36192), state=TIMED_WAITING, group=TGRP-AliasIntegrationTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
	at __randomizedtesting.SeedInfo.seed([7C4E9CA1B8B0EDAC]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.AliasIntegrationTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=3002, name=TEST-AliasIntegrationTest.testDistribSearch-seed#[7C4E9CA1B8B0EDAC]-SendThread(localhost.localdomain:36192), state=TIMED_WAITING, group=TGRP-AliasIntegrationTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=3002, name=TEST-AliasIntegrationTest.testDistribSearch-seed#[7C4E9CA1B8B0EDAC]-SendThread(localhost.localdomain:36192), state=TIMED_WAITING, group=TGRP-AliasIntegrationTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
	at __randomizedtesting.SeedInfo.seed([7C4E9CA1B8B0EDAC]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.BinaryUpdateRequestHandlerTest

Error Message:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:43861 within 30000 ms

Stack Trace:
java.lang.RuntimeException: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:43861 within 30000 ms
	at __randomizedtesting.SeedInfo.seed([7C4E9CA1B8B0EDAC]:0)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:142)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:147)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:98)
	at org.apache.solr.SolrTestCaseJ4.createCore(SolrTestCaseJ4.java:566)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:556)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:365)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:352)
	at org.apache.solr.handler.BinaryUpdateRequestHandlerTest.beforeTests(BinaryUpdateRequestHandlerTest.java:37)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:766)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:43861 within 30000 ms
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:96)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:83)
	at org.apache.solr.cloud.ZkController.checkChrootPath(ZkController.java:667)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:133)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:67)
	at org.apache.solr.core.CoreContainer.load(CoreContainer.java:230)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:137)
	... 29 more
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:43861 within 30000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
	... 37 more


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.BinaryUpdateRequestHandlerTest

Error Message:
ERROR: SolrIndexSearcher opens=0 closes=1

Stack Trace:
java.lang.AssertionError: ERROR: SolrIndexSearcher opens=0 closes=1
	at __randomizedtesting.SeedInfo.seed([7C4E9CA1B8B0EDAC]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.SolrTestCaseJ4.endTrackingSearchers(SolrTestCaseJ4.java:420)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:179)
	at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:789)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.component.DistributedExpandComponentTest

Error Message:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:43861 within 30000 ms

Stack Trace:
java.lang.RuntimeException: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:43861 within 30000 ms
	at __randomizedtesting.SeedInfo.seed([7C4E9CA1B8B0EDAC]:0)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:142)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:147)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:98)
	at org.apache.solr.SolrTestCaseJ4.createCore(SolrTestCaseJ4.java:566)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:556)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:365)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:352)
	at org.apache.solr.handler.component.DistributedExpandComponentTest.setUpBeforeClass(DistributedExpandComponentTest.java:48)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:766)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:43861 within 30000 ms
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:96)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:83)
	at org.apache.solr.cloud.ZkController.checkChrootPath(ZkController.java:667)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:133)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:67)
	at org.apache.solr.core.CoreContainer.load(CoreContainer.java:230)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:137)
	... 29 more
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:43861 within 30000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
	... 37 more


FAILED:  junit.framework.TestSuite.org.apache.solr.search.SpatialFilterTest

Error Message:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:43861 within 30000 ms

Stack Trace:
java.lang.RuntimeException: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:43861 within 30000 ms
	at __randomizedtesting.SeedInfo.seed([7C4E9CA1B8B0EDAC]:0)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:142)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:147)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:98)
	at org.apache.solr.SolrTestCaseJ4.createCore(SolrTestCaseJ4.java:566)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:556)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:365)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:352)
	at org.apache.solr.search.SpatialFilterTest.beforeClass(SpatialFilterTest.java:32)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:766)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:43861 within 30000 ms
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:96)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:83)
	at org.apache.solr.cloud.ZkController.checkChrootPath(ZkController.java:667)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:133)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:67)
	at org.apache.solr.core.CoreContainer.load(CoreContainer.java:230)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:137)
	... 29 more
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:43861 within 30000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
	... 37 more


FAILED:  junit.framework.TestSuite.org.apache.solr.search.TestValueSourceCache

Error Message:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:43861 within 30000 ms

Stack Trace:
java.lang.RuntimeException: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:43861 within 30000 ms
	at __randomizedtesting.SeedInfo.seed([7C4E9CA1B8B0EDAC]:0)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:142)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:147)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:98)
	at org.apache.solr.SolrTestCaseJ4.createCore(SolrTestCaseJ4.java:566)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:556)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:365)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:352)
	at org.apache.solr.search.TestValueSourceCache.beforeClass(TestValueSourceCache.java:32)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:766)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:43861 within 30000 ms
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:96)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:83)
	at org.apache.solr.cloud.ZkController.checkChrootPath(ZkController.java:667)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:133)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:67)
	at org.apache.solr.core.CoreContainer.load(CoreContainer.java:230)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:137)
	... 29 more
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:43861 within 30000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
	... 37 more


FAILED:  junit.framework.TestSuite.org.apache.solr.update.DocumentBuilderTest

Error Message:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:43861 within 30000 ms

Stack Trace:
java.lang.RuntimeException: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:43861 within 30000 ms
	at __randomizedtesting.SeedInfo.seed([7C4E9CA1B8B0EDAC]:0)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:142)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:147)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:98)
	at org.apache.solr.SolrTestCaseJ4.createCore(SolrTestCaseJ4.java:566)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:556)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:365)
	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:352)
	at org.apache.solr.update.DocumentBuilderTest.beforeClass(DocumentBuilderTest.java:54)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:766)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:43861 within 30000 ms
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:96)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:83)
	at org.apache.solr.cloud.ZkController.checkChrootPath(ZkController.java:667)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:133)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:67)
	at org.apache.solr.core.CoreContainer.load(CoreContainer.java:230)
	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:137)
	... 29 more
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:43861 within 30000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
	... 37 more


REGRESSION:  org.apache.solr.cloud.AliasIntegrationTest.testDistribSearch

Error Message:


Stack Trace:
org.apache.solr.common.cloud.ZooKeeperException: 
	at __randomizedtesting.SeedInfo.seed([7C4E9CA1B8B0EDAC:FDA812B9CFEF8D90]:0)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.connect(CloudSolrServer.java:249)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.initCloud(AbstractFullDistribZkTestBase.java:256)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:312)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:868)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:826)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:862)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /live_nodes
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:259)
	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:256)
	at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:299)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.connect(CloudSolrServer.java:242)
	... 42 more


FAILED:  org.apache.solr.rest.schema.TestFieldTypeCollectionResource.testGetAllFieldTypes

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
	at __randomizedtesting.SeedInfo.seed([7C4E9CA1B8B0EDAC:18CB945F696FB6E4]:0)
	at org.apache.solr.util.RestTestBase.assertQ(RestTestBase.java:153)
	at org.apache.solr.rest.schema.TestFieldTypeCollectionResource.testGetAllFieldTypes(TestFieldTypeCollectionResource.java:25)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:826)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:862)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=(/response/arr[@name='fieldTypes']/lst/str[@name='name'])[1] = 'HTMLstandardtok'
	xml response was: <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 503 Server is shutting down or failed to initialize</title>
</head>
<body>
<h2>HTTP ERROR: 503</h2>
<p>Problem accessing /solr/schema/fieldtypes. Reason:
<pre>    Server is shutting down or failed to initialize</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
</body>
</html>

	request was:/schema/fieldtypes?indent=on&wt=xml
	at org.apache.solr.util.RestTestBase.assertQ(RestTestBase.java:146)
	... 40 more


FAILED:  org.apache.solr.rest.schema.TestFieldTypeCollectionResource.testJsonGetAllFieldTypes

Error Message:
JSON Parse Error: char=<,position=0 BEFORE='<' AFTER='html> <head> <meta http-equiv="Content-'

Stack Trace:
org.noggit.JSONParser$ParseException: JSON Parse Error: char=<,position=0 BEFORE='<' AFTER='html> <head> <meta http-equiv="Content-'
	at __randomizedtesting.SeedInfo.seed([7C4E9CA1B8B0EDAC:8B968B36ADB3C828]:0)
	at org.noggit.JSONParser.err(JSONParser.java:223)
	at org.noggit.JSONParser.next(JSONParser.java:622)
	at org.noggit.JSONParser.nextEvent(JSONParser.java:663)
	at org.noggit.ObjectBuilder.<init>(ObjectBuilder.java:44)
	at org.noggit.ObjectBuilder.getVal(ObjectBuilder.java:37)
	at org.noggit.ObjectBuilder.fromJSON(ObjectBuilder.java:33)
	at org.apache.solr.JSONTestUtil.match(JSONTestUtil.java:81)
	at org.apache.solr.JSONTestUtil.match(JSONTestUtil.java:71)
	at org.apache.solr.util.RestTestBase.assertJQ(RestTestBase.java:244)
	at org.apache.solr.util.RestTestBase.assertJQ(RestTestBase.java:196)
	at org.apache.solr.rest.schema.TestFieldTypeCollectionResource.testJsonGetAllFieldTypes(TestFieldTypeCollectionResource.java:33)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:826)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:862)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 10390 lines...]
   [junit4] Suite: org.apache.solr.handler.BinaryUpdateRequestHandlerTest
   [junit4]   2> 473354 T2408 oas.SolrTestCaseJ4.startTrackingSearchers WARN startTrackingSearchers: numOpens=6 numCloses=5
   [junit4]   2> 473355 T2408 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 473355 T2408 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-BinaryUpdateRequestHandlerTest-1395219620167
   [junit4]   2> 473356 T2408 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 473356 T2408 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 473356 T2408 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 473357 T2408 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 473395 T2408 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 473421 T2408 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 473422 T2408 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 473428 T2408 oass.IndexSchema.readSchema [null] Schema name=test
   [junit4]   2> 473574 T2408 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 473578 T2408 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 473579 T2408 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 473584 T2408 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 473586 T2408 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 473587 T2408 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 473587 T2408 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 473588 T2408 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 473588 T2408 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 473588 T2408 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 473588 T2408 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 473589 T2408 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 473589 T2408 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr
   [junit4]   2> 473589 T2408 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/'
   [junit4]   2> 473613 T2408 oasc.CoreContainer.<init> New CoreContainer 21963883
   [junit4]   2> 473614 T2408 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/]
   [junit4]   2> 473614 T2408 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 473614 T2408 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 473615 T2408 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 473615 T2408 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 473615 T2408 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 473615 T2408 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 473615 T2408 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 473616 T2408 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 473616 T2408 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 473618 T2408 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 473618 T2408 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 473618 T2408 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 473618 T2408 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43861/solr
   [junit4]   2> 473619 T2408 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 473619 T2408 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 473620 T2409 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 474257 T2358 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 474536 T2310 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 474536 T2310 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 474536 T2310 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 474537 T2310 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 474537 T2310 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 474537 T2310 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 474538 T2310 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 474538 T2310 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 474722 T2409 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> ASYNC  NEW_CORE C1478 name=testSolrCloudCollection_shard1_replica2 org.apache.solr.core.SolrCore@e15357 url=http://127.0.0.1:59010/solr/testSolrCloudCollection_shard1_replica2 node=127.0.0.1:59010_solr C1478_STATE=coll:testSolrCloudCollection core:testSolrCloudCollection_shard1_replica2 props:{state=recovering, base_url=http://127.0.0.1:59010/solr, core=testSolrCloudCollection_shard1_replica2, node_name=127.0.0.1:59010_solr}
   [junit4]   2> 475364 T2326 C1478 P59010 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
   [junit4]   2> 475364 T2326 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 475365 T2326 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:59010/solr/testSolrCloudCollection_shard1_replica2/ shard1
   [junit4]   2> 475365 T2326 oasc.SolrCore.close [testSolrCloudCollection_shard1_replica2]  CLOSING SolrCore org.apache.solr.core.SolrCore@e15357
   [junit4]   2> 475387 T2326 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
   [junit4]   2> 475388 T2326 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 475388 T2326 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 475388 T2326 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 475389 T2326 oasc.SolrCore.closeSearcher [testSolrCloudCollection_shard1_replica2] Closing main searcher on request.
   [junit4]   2> 475390 T2326 oasc.CachingDirectoryFactory.close Closing RAMDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 475390 T2326 oasc.CachingDirectoryFactory.closeCacheValue looking to close data [CachedDir<<refCount=0;path=data;done=false>>]
   [junit4]   2> 475390 T2326 oasc.CachingDirectoryFactory.close Closing directory: data
   [junit4]   2> 475391 T2326 oasc.CachingDirectoryFactory.closeCacheValue looking to close data/index [CachedDir<<refCount=0;path=data/index;done=false>>]
   [junit4]   2> 475391 T2326 oasc.CachingDirectoryFactory.close Closing directory: data/index
   [junit4]   2> 475391 T2326 oascc.SolrZkClient.makePath makePath: /collections/testSolrCloudCollection/leaders/shard1
   [junit4]   2> 475392 T2326 oasc.SolrException.log ERROR There was a problem trying to register as the leader:org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:135)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:297)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 	Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 		at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:208)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:205)
   [junit4]   2> 		at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:205)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:416)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:385)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:372)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:127)
   [junit4]   2> 		at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:122)
   [junit4]   2> 		... 7 more
   [junit4]   2> 	
   [junit4]   2> 475393 T2326 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/testSolrCloudCollection/leader_elect/shard1/election/91437112227856393-core_node2-n_0000000001
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:176)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:173)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:173)
   [junit4]   2> 	at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:71)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:317)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 475393 T2326 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 475823 T2409 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 476925 T2409 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 478027 T2409 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 479128 T2409 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 480229 T2409 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 481331 T2409 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 482432 T2409 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 483535 T2409 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 484636 T2409 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 485737 T2409 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 486839 T2409 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 487940 T2409 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 489042 T2409 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 490143 T2409 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 491244 T2409 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 492346 T2409 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 493448 T2409 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 494550 T2409 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 495651 T2409 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 496753 T2409 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 497854 T2409 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 498956 T2409 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 500058 T2409 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 501159 T2409 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 502261 T2409 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 503362 T2409 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 504566 T2408 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 625580 T2408 oas.SolrTestCaseJ4.endTrackingSearchers ERROR ERROR: SolrIndexSearcher opens=0 closes=1
   [junit4]   2> NOTE: test params are: codec=Lucene41, sim=DefaultSimilarity, locale=hu_HU, timezone=Africa/Lubumbashi
   [junit4]   2> NOTE: Linux 3.8.0-37-generic i386/Oracle Corporation 1.8.0 (32-bit)/cpus=8,threads=1,free=61299000,total=125108224
   [junit4]   2> NOTE: All tests run in this JVM: [RemoteQueryErrorTest, AnalysisErrorHandlingTest, TestDistribDocBasedVersion, EchoParamsTest, TestReplicationHandler, StatelessScriptUpdateProcessorFactoryTest, PolyFieldTest, TestJoin, TestLMJelinekMercerSimilarityFactory, TestFaceting, SampleTest, SOLR749Test, ScriptEngineTest, BasicDistributedZkTest, SyncSliceTest, ClusterStateTest, TestCoreContainer, DistributedSuggestComponentTest, TestStressUserVersions, AbstractAnalyticsStatsTest, TestCharFilters, DistributedTermsComponentTest, AutoCommitTest, IndexSchemaTest, SearchHandlerTest, TestLMDirichletSimilarityFactory, PeerSyncTest, UpdateParamsTest, QueryEqualityTest, RecoveryZkTest, TestGroupingSearch, LegacyHTMLStripCharFilterTest, TestJmxMonitoredMap, HdfsChaosMonkeySafeLeaderTest, TestDocumentBuilder, CopyFieldTest, FieldFacetExtrasTest, DeleteReplicaTest, CoreAdminHandlerTest, CursorPagingTest, TestStandardQParsers, BinaryUpdateRequestHandlerTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=BinaryUpdateRequestHandlerTest -Dtests.seed=7C4E9CA1B8B0EDAC -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=hu_HU -Dtests.timezone=Africa/Lubumbashi -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J0 | BinaryUpdateRequestHandlerTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.RuntimeException: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:43861 within 30000 ms
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7C4E9CA1B8B0EDAC]:0)
   [junit4]    > 	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:142)
   [junit4]    > 	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:147)
   [junit4]    > 	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:98)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.createCore(SolrTestCaseJ4.java:566)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:556)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:365)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:352)
   [junit4]    > 	at org.apache.solr.handler.BinaryUpdateRequestHandlerTest.beforeTests(BinaryUpdateRequestHandlerTest.java:37)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]    > Caused by: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:43861 within 30000 ms
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:96)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:83)
   [junit4]    > 	at org.apache.solr.cloud.ZkController.checkChrootPath(ZkController.java:667)
   [junit4]    > 	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:133)
   [junit4]    > 	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:67)
   [junit4]    > 	at org.apache.solr.core.CoreContainer.load(CoreContainer.java:230)
   [junit4]    > 	at org.apache.solr.util.TestHarness.<init>(TestHarness.java:137)
   [junit4]    > 	... 29 more
   [junit4]    > Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:43861 within 30000 ms
   [junit4]    > 	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
   [junit4]    > 	... 37 moreThrowable #2: java.lang.AssertionError: ERROR: SolrIndexSearcher opens=0 closes=1
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7C4E9CA1B8B0EDAC]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.endTrackingSearchers(SolrTestCaseJ4.java:420)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:179)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4] Completed on J0 in 152.29s, 0 tests, 1 failure, 1 error <<< FAILURES!

[...truncated 47 lines...]
   [junit4] Suite: org.apache.solr.handler.component.DistributedExpandComponentTest
   [junit4]   2> 625649 T3 oasc.CoreContainer.finalize ERROR CoreContainer was not shutdown prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!  instance=21963883
   [junit4]   2> 625677 T2412 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 625677 T2412 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /_btu/t
   [junit4]   2> 625678 T2412 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-DistributedExpandComponentTest-1395219772490
   [junit4]   2> 625678 T2412 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 625678 T2412 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 625679 T2412 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 625679 T2412 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 625708 T2412 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 625744 T2412 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-collapseqparser.xml
   [junit4]   2> 625744 T2412 oass.IndexSchema.readSchema Reading Solr Schema from schema11.xml
   [junit4]   2> 625749 T2412 oass.IndexSchema.readSchema [null] Schema name=example
   [junit4]   2> 625799 T2412 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 625800 T2412 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 625804 T2412 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 625805 T2412 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr
   [junit4]   2> 625805 T2412 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/'
   [junit4]   2> 625822 T2412 oasc.CoreContainer.<init> New CoreContainer 31880419
   [junit4]   2> 625823 T2412 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/]
   [junit4]   2> 625823 T2412 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 625823 T2412 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 625824 T2412 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 625824 T2412 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 625824 T2412 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 625824 T2412 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 625824 T2412 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 625825 T2412 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 625825 T2412 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 625825 T2412 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 625825 T2412 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 625826 T2412 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 625826 T2412 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43861/solr
   [junit4]   2> 625826 T2412 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 625827 T2412 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 625828 T2413 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 626929 T2413 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 628031 T2413 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 629132 T2413 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 630234 T2413 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 631335 T2413 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 632437 T2413 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.a

[...truncated too long message...]

ead.run WARN Session 0x144d99874b60004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 931019 T3002 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 931020 T3002 oaz.ClientCnxn$SendThread.run WARN Session 0x144d99874b60004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 931519 T3002 oaz.ClientCnxn$SendThread.run WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   [junit4]   2> 
   [junit4]   2> 932020 T3002 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 932021 T3002 oaz.ClientCnxn$SendThread.run WARN Session 0x144d99874b60004 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 932520 T3002 oaz.ClientCnxn$SendThread.run WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   [junit4]   2> 
   [junit4]   2> 28715 T2978 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
   [junit4]   2> 	   1) Thread[id=3002, name=TEST-AliasIntegrationTest.testDistribSearch-seed#[7C4E9CA1B8B0EDAC]-SendThread(localhost.localdomain:36192), state=TIMED_WAITING, group=TGRP-AliasIntegrationTest]
   [junit4]   2> 	        at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> NOTE: test params are: codec=Asserting, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=nl, timezone=Pacific/Kwajalein
   [junit4]   2> NOTE: Linux 3.8.0-37-generic i386/Oracle Corporation 1.8.0 (32-bit)/cpus=8,threads=2,free=55699624,total=125108224
   [junit4]   2> NOTE: All tests run in this JVM: [RemoteQueryErrorTest, AnalysisErrorHandlingTest, TestDistribDocBasedVersion, EchoParamsTest, TestReplicationHandler, StatelessScriptUpdateProcessorFactoryTest, PolyFieldTest, TestJoin, TestLMJelinekMercerSimilarityFactory, TestFaceting, SampleTest, SOLR749Test, ScriptEngineTest, BasicDistributedZkTest, SyncSliceTest, ClusterStateTest, TestCoreContainer, DistributedSuggestComponentTest, TestStressUserVersions, AbstractAnalyticsStatsTest, TestCharFilters, DistributedTermsComponentTest, AutoCommitTest, IndexSchemaTest, SearchHandlerTest, TestLMDirichletSimilarityFactory, PeerSyncTest, UpdateParamsTest, QueryEqualityTest, RecoveryZkTest, TestGroupingSearch, LegacyHTMLStripCharFilterTest, TestJmxMonitoredMap, HdfsChaosMonkeySafeLeaderTest, TestDocumentBuilder, CopyFieldTest, FieldFacetExtrasTest, DeleteReplicaTest, CoreAdminHandlerTest, CursorPagingTest, TestStandardQParsers, BinaryUpdateRequestHandlerTest, DistributedExpandComponentTest, TestValueSourceCache, OverseerTest, DocumentBuilderTest, TestRecoveryHdfs, TestFieldTypeCollectionResource, SpatialFilterTest, OverseerCollectionProcessorTest, AliasIntegrationTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=AliasIntegrationTest -Dtests.seed=7C4E9CA1B8B0EDAC -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=nl -Dtests.timezone=Pacific/Kwajalein -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J0 | AliasIntegrationTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ERROR: SolrZkClient opens=5 closes=4
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7C4E9CA1B8B0EDAC]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:435)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:180)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.AliasIntegrationTest: 
   [junit4]    >    1) Thread[id=3003, name=TEST-AliasIntegrationTest.testDistribSearch-seed#[7C4E9CA1B8B0EDAC]-EventThread, state=WAITING, group=TGRP-AliasIntegrationTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
   [junit4]    >    2) Thread[id=3002, name=TEST-AliasIntegrationTest.testDistribSearch-seed#[7C4E9CA1B8B0EDAC]-SendThread(localhost.localdomain:36192), state=TIMED_WAITING, group=TGRP-AliasIntegrationTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7C4E9CA1B8B0EDAC]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=3002, name=TEST-AliasIntegrationTest.testDistribSearch-seed#[7C4E9CA1B8B0EDAC]-SendThread(localhost.localdomain:36192), state=TIMED_WAITING, group=TGRP-AliasIntegrationTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7C4E9CA1B8B0EDAC]:0)
   [junit4] Completed on J0 in 28.76s, 1 test, 1 failure, 3 errors <<< FAILURES!

[...truncated 747 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1275: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 391 suites, 1622 tests, 9 suite-level errors, 3 errors, 897 ignored (26 assumptions)

Total time: 54 minutes 35 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.8.0 -client -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0) - Build # 9843 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9843/
Java: 32bit/jdk1.8.0 -server -XX:+UseConcMarkSweepGC

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestModifyConfFiles

Error Message:
ERROR: SolrZkClient opens=4 closes=3

Stack Trace:
java.lang.AssertionError: ERROR: SolrZkClient opens=4 closes=3
	at __randomizedtesting.SeedInfo.seed([229540E9FC128DC3]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:435)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:180)
	at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:789)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestModifyConfFiles

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.TestModifyConfFiles:     1) Thread[id=7319, name=TEST-TestModifyConfFiles.testDistribSearch-seed#[229540E9FC128DC3]-EventThread, state=WAITING, group=TGRP-TestModifyConfFiles]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)    2) Thread[id=7318, name=TEST-TestModifyConfFiles.testDistribSearch-seed#[229540E9FC128DC3]-SendThread(localhost.localdomain:59210), state=TIMED_WAITING, group=TGRP-TestModifyConfFiles]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.TestModifyConfFiles: 
   1) Thread[id=7319, name=TEST-TestModifyConfFiles.testDistribSearch-seed#[229540E9FC128DC3]-EventThread, state=WAITING, group=TGRP-TestModifyConfFiles]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
   2) Thread[id=7318, name=TEST-TestModifyConfFiles.testDistribSearch-seed#[229540E9FC128DC3]-SendThread(localhost.localdomain:59210), state=TIMED_WAITING, group=TGRP-TestModifyConfFiles]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
	at __randomizedtesting.SeedInfo.seed([229540E9FC128DC3]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestModifyConfFiles

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=7318, name=TEST-TestModifyConfFiles.testDistribSearch-seed#[229540E9FC128DC3]-SendThread(localhost.localdomain:59210), state=TIMED_WAITING, group=TGRP-TestModifyConfFiles]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=7318, name=TEST-TestModifyConfFiles.testDistribSearch-seed#[229540E9FC128DC3]-SendThread(localhost.localdomain:59210), state=TIMED_WAITING, group=TGRP-TestModifyConfFiles]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
	at __randomizedtesting.SeedInfo.seed([229540E9FC128DC3]:0)


REGRESSION:  org.apache.solr.cloud.TestModifyConfFiles.testDistribSearch

Error Message:


Stack Trace:
org.apache.solr.common.cloud.ZooKeeperException: 
	at __randomizedtesting.SeedInfo.seed([229540E9FC128DC3:A373CEF18B4DEDFF]:0)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.connect(CloudSolrServer.java:249)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.initCloud(AbstractFullDistribZkTestBase.java:256)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:312)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:868)
	at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:826)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:862)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /live_nodes
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:259)
	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:256)
	at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:299)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.connect(CloudSolrServer.java:242)
	... 41 more




Build Log:
[...truncated 11235 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestModifyConfFiles
   [junit4]   2> 1500036 T7297 oas.SolrTestCaseJ4.startTrackingSearchers WARN startTrackingSearchers: numOpens=6 numCloses=6
   [junit4]   2> 1500036 T7297 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 1500037 T7297 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /juzl/oi
   [junit4]   2> 1500040 T7297 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestModifyConfFiles-1395215080199
   [junit4]   2> 1500040 T7297 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1500041 T7298 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1500141 T7297 oasc.ZkTestServer.run start zk server on port:59210
   [junit4]   2> 1500142 T7297 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1500145 T7304 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1047487 name:ZooKeeperConnection Watcher:127.0.0.1:59210 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1500146 T7297 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1500146 T7297 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1500152 T7297 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1500153 T7306 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@145de0d name:ZooKeeperConnection Watcher:127.0.0.1:59210/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1500153 T7297 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1500153 T7297 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1500157 T7297 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1500161 T7297 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1500165 T7297 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1500170 T7297 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1500170 T7297 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1500176 T7297 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1500177 T7297 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1500182 T7297 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1500182 T7297 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1500186 T7297 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1500187 T7297 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1500191 T7297 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1500192 T7297 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1500196 T7297 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1500197 T7297 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1500200 T7297 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1500201 T7297 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1500203 T7297 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1500203 T7297 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1500205 T7297 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1500205 T7297 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1500207 T7297 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1500208 T7297 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1500212 T7297 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1500213 T7297 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1500433 T7297 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1500437 T7297 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60519
   [junit4]   2> 1500437 T7297 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1500438 T7297 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1500439 T7297 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.TestModifyConfFiles-controljetty-1395215080377
   [junit4]   2> 1500439 T7297 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.TestModifyConfFiles-controljetty-1395215080377/'
   [junit4]   2> 1500451 T7297 oass.SolrDispatchFilter.loadConfigSolr Trying to read solr.xml from 127.0.0.1:59210/solr
   [junit4]   2> 1500452 T7297 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1500453 T7317 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@133daa5 name:ZooKeeperConnection Watcher:127.0.0.1:59210/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1500453 T7297 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1500456 T7297 oass.SolrDispatchFilter.init ERROR Could not start Solr. Check solr/home property and the logs
   [junit4]   2> 1500457 T7297 oasc.SolrException.log ERROR null:org.apache.solr.common.SolrException: Could not load solr.xml from zookeeper
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.loadConfigSolr(SolrDispatchFilter.java:182)
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:198)
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
   [junit4]   2> 		at org.eclipse.jetty.servlet.FilterHolder.doStart(FilterHolder.java:119)
   [junit4]   2> 		at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:719)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1309)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1345)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1085)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:931)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:888)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:340)
   [junit4]   2> 		at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:325)
   [junit4]   2> 		at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:174)
   [junit4]   2> 		at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:65)
   [junit4]   2> 		at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:430)
   [junit4]   2> 		at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:403)
   [junit4]   2> 		at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJetty(AbstractFullDistribZkTestBase.java:490)
   [junit4]   2> 		at org.apache.solr.BaseDistributedSearchTestCase.createJetty(BaseDistributedSearchTestCase.java:356)
   [junit4]   2> 		at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:291)
   [junit4]   2> 		at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:868)
   [junit4]   2> 		at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source)
   [junit4]   2> 		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 		at java.lang.reflect.Method.invoke(Method.java:483)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:826)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:862)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
   [junit4]   2> 		at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 		at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
   [junit4]   2> 		at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	Caused by: org.apache.solr.common.SolrException: Could not load solr.xml from zookeeper: node not found
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.loadConfigSolr(SolrDispatchFilter.java:178)
   [junit4]   2> 		... 58 more
   [junit4]   2> 	
   [junit4]   2> 1500458 T7297 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1500460 T7297 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1500461 T7319 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1130eb3 name:ZooKeeperConnection Watcher:127.0.0.1:59210/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1500462 T7297 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1500463 T7297 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1500466 T7297 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1500469 T7297 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1500471 T7297 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 60519
   [junit4]   2> 1500493 T7297 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/juzl/oi,null}
   [junit4]   2> 1500547 T7297 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 1500550 T7297 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:59210 59210
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestModifyConfFiles -Dtests.method=testDistribSearch -Dtests.seed=229540E9FC128DC3 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ko -Dtests.timezone=GB-Eire -Dtests.file.encoding=UTF-8
   [junit4] ERROR   10.7s J0 | TestModifyConfFiles.testDistribSearch <<<
   [junit4]    > Throwable #1: org.apache.solr.common.cloud.ZooKeeperException: 
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([229540E9FC128DC3:A373CEF18B4DEDFF]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.connect(CloudSolrServer.java:249)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.initCloud(AbstractFullDistribZkTestBase.java:256)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:312)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:868)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]    > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /live_nodes
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]    > 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:259)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:256)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:299)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.connect(CloudSolrServer.java:242)
   [junit4]    > 	... 41 more
   [junit4]   2> 1510766 T7297 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 1510767 T7297 oas.SolrTestCaseJ4.endTrackingZkClients ERROR ERROR: SolrZkClient opens=4 closes=3
   [junit4]   2> 10734 T7296 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> 1510844 T7319 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1130eb3 name:ZooKeeperConnection Watcher:127.0.0.1:59210/solr got event WatchedEvent state:Disconnected type:None path:null path:null type:None
   [junit4]   2> 1510845 T7319 oascc.ConnectionManager.process zkClient has disconnected
   [junit4]   2> 1512272 T7318 oaz.ClientCnxn$SendThread.run WARN Session 0x144d94c9b0a0003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1514033 T7318 oaz.ClientCnxn$SendThread.run WARN Session 0x144d94c9b0a0003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1515752 T7318 oaz.ClientCnxn$SendThread.run WARN Session 0x144d94c9b0a0003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1517804 T7318 oaz.ClientCnxn$SendThread.run WARN Session 0x144d94c9b0a0003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1519680 T7318 oaz.ClientCnxn$SendThread.run WARN Session 0x144d94c9b0a0003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1521055 T7318 oaz.ClientCnxn$SendThread.run WARN Session 0x144d94c9b0a0003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1522876 T7318 oaz.ClientCnxn$SendThread.run WARN Session 0x144d94c9b0a0003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1524290 T7318 oaz.ClientCnxn$SendThread.run WARN Session 0x144d94c9b0a0003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1525396 T7318 oaz.ClientCnxn$SendThread.run WARN Session 0x144d94c9b0a0003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1527099 T7318 oaz.ClientCnxn$SendThread.run WARN Session 0x144d94c9b0a0003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1528249 T7318 oaz.ClientCnxn$SendThread.run WARN Session 0x144d94c9b0a0003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1529396 T7318 oaz.ClientCnxn$SendThread.run WARN Session 0x144d94c9b0a0003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 30870 T7296 ccr.ThreadLeakControl.checkThreadLeaks SEVERE 2 threads leaked from SUITE scope at org.apache.solr.cloud.TestModifyConfFiles: 
   [junit4]   2> 	   1) Thread[id=7319, name=TEST-TestModifyConfFiles.testDistribSearch-seed#[229540E9FC128DC3]-EventThread, state=WAITING, group=TGRP-TestModifyConfFiles]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]   2> 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
   [junit4]   2> 	   2) Thread[id=7318, name=TEST-TestModifyConfFiles.testDistribSearch-seed#[229540E9FC128DC3]-SendThread(localhost.localdomain:59210), state=TIMED_WAITING, group=TGRP-TestModifyConfFiles]
   [junit4]   2> 	        at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 30871 T7296 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads:
   [junit4]   2> 	   1) Thread[id=7319, name=TEST-TestModifyConfFiles.testDistribSearch-seed#[229540E9FC128DC3]-EventThread, state=WAITING, group=TGRP-TestModifyConfFiles]
   [junit4]   2> 	   2) Thread[id=7318, name=TEST-TestModifyConfFiles.testDistribSearch-seed#[229540E9FC128DC3]-SendThread(localhost.localdomain:59210), state=TIMED_WAITING, group=TGRP-TestModifyConfFiles]
   [junit4]   2> 1530908 T7318 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 1530907 T7319 oaz.ClientCnxn$EventThread.run ERROR Event thread exiting due to interruption java.lang.InterruptedException
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
   [junit4]   2> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
   [junit4]   2> 
   [junit4]   2> 1530909 T7318 oaz.ClientCnxn$SendThread.run WARN Session 0x144d94c9b0a0003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1531407 T7318 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 1531408 T7318 oaz.ClientCnxn$SendThread.run WARN Session 0x144d94c9b0a0003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1531908 T7318 oaz.ClientCnxn$SendThread.run WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   [junit4]   2> 
   [junit4]   2> 1532408 T7318 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 1532409 T7318 oaz.ClientCnxn$SendThread.run WARN Session 0x144d94c9b0a0003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1532908 T7318 oaz.ClientCnxn$SendThread.run WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   [junit4]   2> 
   [junit4]   2> 1533409 T7318 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 1533410 T7318 oaz.ClientCnxn$SendThread.run WARN Session 0x144d94c9b0a0003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 33873 T7296 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
   [junit4]   2> 	   1) Thread[id=7318, name=TEST-TestModifyConfFiles.testDistribSearch-seed#[229540E9FC128DC3]-SendThread(localhost.localdomain:59210), state=TIMED_WAITING, group=TGRP-TestModifyConfFiles]
   [junit4]   2> 	        at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> NOTE: test params are: codec=Lucene41, sim=DefaultSimilarity, locale=ko, timezone=GB-Eire
   [junit4]   2> NOTE: Linux 3.8.0-37-generic i386/Oracle Corporation 1.8.0 (32-bit)/cpus=8,threads=2,free=57564480,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [CachingDirectoryFactoryTest, CSVRequestHandlerTest, TestFieldTypeResource, DOMUtilTest, TestDocumentBuilder, TestFieldResource, DistributedTermsComponentTest, SampleTest, TestComplexPhraseQParserPlugin, BinaryUpdateRequestHandlerTest, SOLR749Test, TestLuceneMatchVersion, QueryElevationComponentTest, TestPostingsSolrHighlighter, TriLevelCompositeIdRoutingTest, BlockCacheTest, InfoHandlerTest, FunctionTest, BlockDirectoryTest, TermsComponentTest, TestExceedMaxTermLength, TestSolrXMLSerializer, TestNonNRTOpen, TestSchemaSimilarityResource, LoggingHandlerTest, TestOverriddenPrefixQueryForCustomFieldType, StatelessScriptUpdateProcessorFactoryTest, TestRecoveryHdfs, TestLMJelinekMercerSimilarityFactory, TestQueryUtils, ExternalFileFieldSortTest, TestPHPSerializedResponseWriter, TestInitQParser, TestFunctionQuery, DeleteInactiveReplicaTest, BadComponentTest, DocValuesMissingTest, AnalysisAfterCoreReloadTest, TestPerFieldSimilarity, TestCollationFieldDocValues, DistributedSpellCheckComponentTest, AbstractAnalyticsStatsTest, TestSuggestSpellingConverter, TestGroupingSearch, DistributedQueryElevationComponentTest, NoFacetTest, JsonLoaderTest, TestCoreContainer, ClusterStateUpdateTest, HdfsDirectoryTest, OverseerTest, TermVectorComponentTest, SolrTestCaseJ4Test, TestFreeTextSuggestions, SuggesterFSTTest, TestReplicationHandler, LukeRequestHandlerTest, TestConfig, SoftAutoCommitTest, ZkCLITest, SuggestComponentTest, SolrIndexSplitterTest, DistributedExpandComponentTest, FastVectorHighlighterTest, FieldMutatingUpdateProcessorTest, TestDistributedMissingSort, DeleteShardTest, ModifyConfFileTest, TestSolrIndexConfig, EnumFieldTest, BasicFunctionalityTest, TestSearchPerf, SchemaVersionSpecificBehaviorTest, URLClassifyProcessorTest, TestSolrQueryParser, TestCloudManagedSchemaAddField, SSLMigrationTest, TestExtendedDismaxParser, CacheHeaderTest, MoreLikeThisHandlerTest, FieldFacetExtrasTest, ClusterStateTest, TestAddFieldRealTimeGet, SystemInfoHandlerTest, SearchHandlerTest, LeaderElectionTest, RemoteQueryErrorTest, CursorPagingTest, DirectUpdateHandlerOptimizeTest, QueryParsingTest, StressHdfsTest, TestElisionMultitermQuery, CopyFieldTest, SyncSliceTest, TestStressVersions, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, CollectionsAPIDistributedZkTest, TestRandomDVFaceting, SliceStateUpdateTest, TestDistributedGrouping, TermVectorComponentDistributedTest, TestRealTimeGet, TestJoin, SimpleFacetsTest, StatsComponentTest, SolrCmdDistributorTest, ConvertedLegacyTest, TestLazyCores, TestBadConfig, TestIndexSearcher, ShowFileRequestHandlerTest, TestCoreDiscovery, DocValuesTest, SuggesterTSTTest, SuggesterTest, SpatialFilterTest, TestCSVLoader, NoCacheHeaderTest, WordBreakSolrSpellCheckerTest, TestAtomicUpdateErrorCases, QueryEqualityTest, TestRemoteStreaming, TestSolrDeletionPolicy1, IndexBasedSpellCheckerTest, DirectSolrSpellCheckerTest, TestReversedWildcardFilterFactory, PrimitiveFieldTypeTest, XmlUpdateRequestHandlerTest, RequestHandlersTest, PathHierarchyTokenizerFactoryTest, RequiredFieldsTest, TestArbitraryIndexDir, IndexSchemaTest, TestCollationField, UpdateRequestProcessorFactoryTest, MBeansHandlerTest, TestComponentsName, CoreContainerCoreInitFailuresTest, HighlighterConfigTest, UpdateParamsTest, ResponseLogComponentTest, TestMergePolicyConfig, MultiTermTest, TestDocSet, TestBinaryField, NumericFieldsTest, MinimalSchemaTest, TestSolrCoreProperties, DirectSolrConnectionTest, NotRequiredUniqueKeyTest, SpellPossibilityIteratorTest, TestCharFilters, SynonymTokenizerTest, EchoParamsTest, TestIBSimilarityFactory, TestDefaultSimilarityFactory, LegacyHTMLStripCharFilterTest, TimeZoneUtilsTest, ResourceLoaderTest, ScriptEngineTest, DateMathParserTest, PreAnalyzedFieldTest, TestSystemIdResolver, PrimUtilsTest, DateFieldTest, TestLRUCache, TestUtils, ZkNodePropsTest, SliceStateTest, UUIDFieldTest, CircularListTest, TestCursorMarkWithoutUniqueKey, QueryFacetTest, AliasIntegrationTest, AssignTest, AsyncMigrateRouteKeyTest, ConnectionManagerTest, CustomCollectionTest, DeleteReplicaTest, DistribCursorPagingTest, MigrateRouteKeyTest, OverseerRolesTest, TestDistribDocBasedVersion, TestModifyConfFiles]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestModifyConfFiles -Dtests.seed=229540E9FC128DC3 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ko -Dtests.timezone=GB-Eire -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J0 | TestModifyConfFiles (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ERROR: SolrZkClient opens=4 closes=3
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([229540E9FC128DC3]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:435)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:180)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.TestModifyConfFiles: 
   [junit4]    >    1) Thread[id=7319, name=TEST-TestModifyConfFiles.testDistribSearch-seed#[229540E9FC128DC3]-EventThread, state=WAITING, group=TGRP-TestModifyConfFiles]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
   [junit4]    >    2) Thread[id=7318, name=TEST-TestModifyConfFiles.testDistribSearch-seed#[229540E9FC128DC3]-SendThread(localhost.localdomain:59210), state=TIMED_WAITING, group=TGRP-TestModifyConfFiles]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([229540E9FC128DC3]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=7318, name=TEST-TestModifyConfFiles.testDistribSearch-seed#[229540E9FC128DC3]-SendThread(localhost.localdomain:59210), state=TIMED_WAITING, group=TGRP-TestModifyConfFiles]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([229540E9FC128DC3]:0)
   [junit4] Completed on J0 in 33.92s, 1 test, 1 failure, 3 errors <<< FAILURES!

[...truncated 98 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1275: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 391 suites, 1645 tests, 3 suite-level errors, 1 error, 120 ignored (28 assumptions)

Total time: 55 minutes 15 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.8.0 -server -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0) - Build # 9842 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9842/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseG1GC

All tests passed

Build Log:
[...truncated 30970 lines...]
-check-forbidden-tests:
[forbidden-apis] Reading API signatures: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/tools/forbiddenApis/tests.txt
[forbidden-apis] Loading classes to check...
[forbidden-apis] Scanning for API signatures and dependencies...
[forbidden-apis] Forbidden method invocation: java.util.Random#<init>() [Use RandomizedRunner's random instead]
[forbidden-apis]   in org.apache.solr.cloud.TestMiniSolrCloudCluster (TestMiniSolrCloudCluster.java:67)
[forbidden-apis] Scanned 656 (and 914 related) class file(s) for forbidden API invocations (in 0.39s), 1 error(s).

BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:70: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:271: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:2231: Check for forbidden API calls failed, see log.

Total time: 56 minutes 44 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Re: [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_60-ea-b10) - Build # 9841 - Failure!

Posted by "Smiley, David W." <ds...@mitre.org>.
Hmmm; I’ll fix this ASAP


On 3/18/14, 11:53 PM, "Policeman Jenkins Server" <je...@thetaphi.de>
wrote:

>Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9841/
>Java: 32bit/jdk1.7.0_60-ea-b10 -client -XX:+UseG1GC
>
>1 tests failed.
>FAILED:  
>org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.testWith
>in {#9 seed=[14A074E14CFDBEA3:F66F35851492BBDC]}
>
>Error Message:
>Shouldn't match I#3:Rect(minX=-166.0,maxX=-161.0,minY=87.0,maxY=90.0)
>Q:Pt(x=180.0,y=56.0)
>
>Stack Trace:
>java.lang.AssertionError: Shouldn't match
>I#3:Rect(minX=-166.0,maxX=-161.0,minY=87.0,maxY=90.0) Q:Pt(x=180.0,y=56.0)
>	at 
>__randomizedtesting.SeedInfo.seed([14A074E14CFDBEA3:F66F35851492BBDC]:0)
>	at org.junit.Assert.fail(Assert.java:93)
>	at 
>org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.fail(Spa
>tialOpRecursivePrefixTreeTest.java:355)
>	at 
>org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.doTest(S
>patialOpRecursivePrefixTreeTest.java:335)
>	at 
>org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.testWith
>in(SpatialOpRecursivePrefixTreeTest.java:119)
>	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>	at 
>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
>57)
>	at 
>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>pl.java:43)
>	at java.lang.reflect.Method.invoke(Method.java:606)
>	at 
>com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunne
>r.java:1617)
>	at 
>com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedR
>unner.java:826)
>	at 
>com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedR
>unner.java:862)
>	at 
>com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedR
>unner.java:876)
>	at 
>org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSet
>upTeardownChained.java:50)
>	at 
>org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCa
>cheSanity.java:51)
>	at 
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAf
>terRule.java:46)
>	at 
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.e
>valuate(SystemPropertiesInvariantRule.java:55)
>	at 
>org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThread
>AndTestName.java:49)
>	at 
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleI
>gnoreAfterMaxFailures.java:70)
>	at 
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.
>java:48)
>	at 
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme
>ntAdapter.java:36)
>	at 
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(T
>hreadLeakControl.java:359)
>	at 
>com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(Th
>readLeakControl.java:783)
>	at 
>com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeak
>Control.java:443)
>	at 
>com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(Randomiz
>edRunner.java:835)
>	at 
>com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedR
>unner.java:737)
>	at 
>com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedR
>unner.java:771)
>	at 
>com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedR
>unner.java:782)
>	at 
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAf
>terRule.java:46)
>	at 
>org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClas
>sName.java:42)
>	at 
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.e
>valuate(SystemPropertiesInvariantRule.java:55)
>	at 
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRu
>le$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at 
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRu
>le$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at 
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme
>ntAdapter.java:36)
>	at 
>org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAsser
>tionsRequired.java:43)
>	at 
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.
>java:48)
>	at 
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleI
>gnoreAfterMaxFailures.java:70)
>	at 
>org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreT
>estSuites.java:55)
>	at 
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme
>ntAdapter.java:36)
>	at 
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(T
>hreadLeakControl.java:359)
>	at java.lang.Thread.run(Thread.java:744)
>
>
>
>
>Build Log:
>[...truncated 9265 lines...]
>   [junit4] Suite:
>org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(GeohashPrefixTree(
>maxLevels:1,ctx:SpatialContext.GEO)))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxL
>evels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(max
>Levels:3,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(GeohashPrefixTree(
>maxLevels:2,ctx:SpatialContext.GEO)))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxL
>evels:4,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(GeohashPrefixTree(
>maxLevels:1,ctx:SpatialContext.GEO)))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(GeohashPrefixTree(
>maxLevels:1,ctx:SpatialContext.GEO)))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(max
>Levels:3,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxL
>evels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(GeohashPrefixTree(
>maxLevels:1,ctx:SpatialContext.GEO)))
>   [junit4]   2> Ig:Rect(minX=-180.0,maxX=-135.0,minY=45.0,maxY=90.0)
>Qg:Rect(minX=135.0,maxX=180.0,minY=45.0,maxY=90.0)
>   [junit4]   2> NOTE: reproduce with: ant test
>-Dtestcase=SpatialOpRecursivePrefixTreeTest -Dtests.method=testWithin
>-Dtests.seed=14A074E14CFDBEA3 -Dtests.multiplier=3 -Dtests.slow=true
>-Dtests.locale=sv -Dtests.timezone=Pacific/Kosrae
>-Dtests.file.encoding=UTF-8
>   [junit4] FAILURE 0.13s J0 |
>SpatialOpRecursivePrefixTreeTest.testWithin {#9
>seed=[14A074E14CFDBEA3:F66F35851492BBDC]} <<<
>   [junit4]    > Throwable #1: java.lang.AssertionError: Shouldn't match
>I#3:Rect(minX=-166.0,maxX=-161.0,minY=87.0,maxY=90.0) Q:Pt(x=180.0,y=56.0)
>   [junit4]    > 	at
>__randomizedtesting.SeedInfo.seed([14A074E14CFDBEA3:F66F35851492BBDC]:0)
>   [junit4]    > 	at
>org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.fail(Spa
>tialOpRecursivePrefixTreeTest.java:355)
>   [junit4]    > 	at
>org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.doTest(S
>patialOpRecursivePrefixTreeTest.java:335)
>   [junit4]    > 	at
>org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.testWith
>in(SpatialOpRecursivePrefixTreeTest.java:119)
>   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(GeohashPrefixTree(
>maxLevels:2,ctx:SpatialContext.GEO)))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(GeohashPrefixTree(
>maxLevels:3,ctx:SpatialContext.GEO)))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxL
>evels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxL
>evels:4,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxL
>evels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxL
>evels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxL
>evels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(GeohashPrefixTree(
>maxLevels:2,ctx:SpatialContext.GEO)))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(GeohashPrefixTree(
>maxLevels:3,ctx:SpatialContext.GEO)))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:1,SPG:(QuadPrefixTree(maxL
>evels:5,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(GeohashPrefixTree(
>maxLevels:3,ctx:SpatialContext.GEO)))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxL
>evels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(max
>Levels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:1,SPG:(QuadPrefixTree(maxL
>evels:5,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxL
>evels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(max
>Levels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(GeohashPrefixTree(
>maxLevels:2,ctx:SpatialContext.GEO)))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxL
>evels:4,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(max
>Levels:3,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxL
>evels:4,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxL
>evels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(GeohashPrefixTree(
>maxLevels:2,ctx:SpatialContext.GEO)))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxL
>evels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxL
>evels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(GeohashPrefixTree(
>maxLevels:3,ctx:SpatialContext.GEO)))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxL
>evels:4,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxL
>evels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(max
>Levels:3,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(max
>Levels:3,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>   [junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(GeohashPrefixTree(
>maxLevels:3,ctx:SpatialContext.GEO)))
>   [junit4]   2> NOTE: test params are:
>codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=Compress
>ingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=225),
>termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMP
>RESSION, chunkSize=225)),
>sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=sv,
>timezone=Pacific/Kosrae
>   [junit4]   2> NOTE: Linux 3.8.0-37-generic i386/Oracle Corporation
>1.7.0_60-ea (32-bit)/cpus=8,threads=1,free=16393832,total=91226112
>   [junit4]   2> NOTE: All tests run in this JVM: [SpatialPrefixTreeTest,
>TestRecursivePrefixTreeStrategy, SpatialOpRecursivePrefixTreeTest]
>   [junit4] Completed on J0 in 1.99s, 43 tests, 1 failure <<< FAILURES!
>
>[...truncated 35 lines...]
>BUILD FAILED
>/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The
>following error occurred while executing this line:
>/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The
>following error occurred while executing this line:
>/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The
>following error occurred while executing this line:
>/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37:
>The following error occurred while executing this line:
>/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build.xml:539:
>The following error occurred while executing this line:
>/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml
>:1996: The following error occurred while executing this line:
>/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/module-build.xml
>:60: The following error occurred while executing this line:
>/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml
>:1275: The following error occurred while executing this line:
>/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml
>:907: There were test failures: 14 suites, 79 tests, 1 failure, 2 ignored
>(2 assumptions)
>
>Total time: 32 minutes 13 seconds
>Build step 'Invoke Ant' marked build as failure
>Description set: Java: 32bit/jdk1.7.0_60-ea-b10 -client -XX:+UseG1GC
>Archiving artifacts
>Recording test results
>Email was triggered for: Failure
>Sending email for trigger: Failure
>
>