You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/07/08 19:03:40 UTC

[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_05) - Build # 10759 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10759/
Java: 64bit/jdk1.8.0_05 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

3 tests failed.
FAILED:  org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations {#4 seed=[D219D8E1E28D9A88:E7A2011F26D509E5]}

Error Message:
[Contains] Shouldn't match I#1:Rect(minX=50.0,maxX=120.0,minY=-90.0,maxY=-90.0) Q:Rect(minX=80.0,maxX=80.0,minY=-90.0,maxY=-90.0)

Stack Trace:
java.lang.AssertionError: [Contains] Shouldn't match I#1:Rect(minX=50.0,maxX=120.0,minY=-90.0,maxY=-90.0) Q:Rect(minX=80.0,maxX=80.0,minY=-90.0,maxY=-90.0)
	at __randomizedtesting.SeedInfo.seed([D219D8E1E28D9A88:E7A2011F26D509E5]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:62)
	at org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations(TestBBoxStrategy.java:97)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations {#9 seed=[D219D8E1E28D9A88:31129F6879D1E045]}

Error Message:
[IsWithin] Shouldn't match I#0:Rect(minX=270.0,maxX=280.0,minY=-80.0,maxY=-80.0) Q:Rect(minX=170.0,maxX=300.0,minY=-80.0,maxY=-80.0)

Stack Trace:
java.lang.AssertionError: [IsWithin] Shouldn't match I#0:Rect(minX=270.0,maxX=280.0,minY=-80.0,maxY=-80.0) Q:Rect(minX=170.0,maxX=300.0,minY=-80.0,maxY=-80.0)
	at __randomizedtesting.SeedInfo.seed([D219D8E1E28D9A88:31129F6879D1E045]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:62)
	at org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations(TestBBoxStrategy.java:97)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations {#10 seed=[D219D8E1E28D9A88:C4537733AB6DC29D]}

Error Message:
[IsWithin] Shouldn't match I#3:Rect(minX=-150.0,maxX=-80.0,minY=-90.0,maxY=-90.0) Q:Rect(minX=40.0,maxX=-60.0,minY=-90.0,maxY=-90.0)

Stack Trace:
java.lang.AssertionError: [IsWithin] Shouldn't match I#3:Rect(minX=-150.0,maxX=-80.0,minY=-90.0,maxY=-90.0) Q:Rect(minX=40.0,maxX=-60.0,minY=-90.0,maxY=-90.0)
	at __randomizedtesting.SeedInfo.seed([D219D8E1E28D9A88:C4537733AB6DC29D]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:62)
	at org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations(TestBBoxStrategy.java:97)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 9631 lines...]
   [junit4] Suite: org.apache.lucene.spatial.bbox.TestBBoxStrategy
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestBBoxStrategy -Dtests.method=testOperations -Dtests.seed=D219D8E1E28D9A88 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=iw_IL -Dtests.timezone=Asia/Seoul -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 0.17s J1 | TestBBoxStrategy.testOperations {#4 seed=[D219D8E1E28D9A88:E7A2011F26D509E5]} <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: [Contains] Shouldn't match I#1:Rect(minX=50.0,maxX=120.0,minY=-90.0,maxY=-90.0) Q:Rect(minX=80.0,maxX=80.0,minY=-90.0,maxY=-90.0)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([D219D8E1E28D9A88:E7A2011F26D509E5]:0)
   [junit4]    > 	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
   [junit4]    > 	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
   [junit4]    > 	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:62)
   [junit4]    > 	at org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations(TestBBoxStrategy.java:97)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestBBoxStrategy -Dtests.method=testOperations -Dtests.seed=D219D8E1E28D9A88 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=iw_IL -Dtests.timezone=Asia/Seoul -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 0.11s J1 | TestBBoxStrategy.testOperations {#9 seed=[D219D8E1E28D9A88:31129F6879D1E045]} <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: [IsWithin] Shouldn't match I#0:Rect(minX=270.0,maxX=280.0,minY=-80.0,maxY=-80.0) Q:Rect(minX=170.0,maxX=300.0,minY=-80.0,maxY=-80.0)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([D219D8E1E28D9A88:31129F6879D1E045]:0)
   [junit4]    > 	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
   [junit4]    > 	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
   [junit4]    > 	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:62)
   [junit4]    > 	at org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations(TestBBoxStrategy.java:97)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestBBoxStrategy -Dtests.method=testOperations -Dtests.seed=D219D8E1E28D9A88 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=iw_IL -Dtests.timezone=Asia/Seoul -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 0.19s J1 | TestBBoxStrategy.testOperations {#10 seed=[D219D8E1E28D9A88:C4537733AB6DC29D]} <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: [IsWithin] Shouldn't match I#3:Rect(minX=-150.0,maxX=-80.0,minY=-90.0,maxY=-90.0) Q:Rect(minX=40.0,maxX=-60.0,minY=-90.0,maxY=-90.0)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([D219D8E1E28D9A88:C4537733AB6DC29D]:0)
   [junit4]    > 	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
   [junit4]    > 	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
   [junit4]    > 	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:62)
   [junit4]    > 	at org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations(TestBBoxStrategy.java:97)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> יול 09, 2014 2:03:24 AM org.apache.lucene.spatial.StrategyTestCase executeQueries
   [junit4]   2> INFO: testing queried for strategy BBoxStrategy field:bbox ctx=SpatialContext.GEO
   [junit4]   2> יול 09, 2014 2:03:24 AM org.apache.lucene.spatial.StrategyTestCase executeQueries
   [junit4]   2> INFO: testing queried for strategy BBoxStrategy field:bbox ctx=SpatialContext.GEO
   [junit4]   2> יול 09, 2014 2:03:24 AM org.apache.lucene.spatial.StrategyTestCase executeQueries
   [junit4]   2> INFO: testing queried for strategy BBoxStrategy field:bbox ctx=SpatialContext.GEO
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build/spatial/test/J1/./temp/lucene.spatial.bbox.TestBBoxStrategy-D219D8E1E28D9A88-001
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {bbox__xdl=LM Jelinek-Mercer(0.100000), bbox__maxX=DFR I(n)2, bbox__maxY=DFR GB3(800.0), bbox__minX=DFR GL3(800.0), bbox__minY=DFR GL1}, locale=iw_IL, timezone=Asia/Seoul
   [junit4]   2> NOTE: Linux 3.8.0-42-generic amd64/Oracle Corporation 1.8.0_05 (64-bit)/cpus=8,threads=1,free=133908160,total=189792256
   [junit4]   2> NOTE: All tests run in this JVM: [SpatialArgsParserTest, SpatialPrefixTreeTest, TestPointVectorStrategy, DateNRStrategyTest, TestBBoxStrategy]
   [junit4] Completed on J1 in 4.90s, 30 tests, 3 failures, 1 skipped <<< FAILURES!

[...truncated 12 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build.xml:544: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:2019: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/module-build.xml:60: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1298: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:922: There were test failures: 16 suites, 143 tests, 3 failures, 13 ignored (2 assumptions)

Total time: 40 minutes 16 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_05 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_60) - Build # 10762 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10762/
Java: 64bit/jdk1.7.0_60 -XX:-UseCompressedOops -XX:+UseParallelGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.OverseerTest.testOverseerFailure

Error Message:
Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed

Stack Trace:
org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
	at __randomizedtesting.SeedInfo.seed([B9E2DD92FCD61EDC:BDEA5261EE73F1FD]:0)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:143)
	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:155)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:314)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:660)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:135)
	at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:130)
	... 47 more




Build Log:
[...truncated 11374 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.OverseerTest-B9E2DD92FCD61EDC-001/init-core-data-001
   [junit4]   2> 939317 T3305 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 939317 T3305 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 939317 T3305 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 939320 T3305 oas.SolrTestCaseJ4.setUp ###Starting testDoubleAssignment
   [junit4]   2> 939320 T3305 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 939321 T3306 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 939421 T3305 oasc.ZkTestServer.run start zk server on port:40069
   [junit4]   2> 939422 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 939424 T3312 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7ff62203 name:ZooKeeperConnection Watcher:127.0.0.1:40069/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 939425 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 939425 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 939426 T3314 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4eaac9a7 name:ZooKeeperConnection Watcher:127.0.0.1:40069 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 939427 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 939430 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 939431 T3316 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6e08d82b name:ZooKeeperConnection Watcher:127.0.0.1:40069 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 939432 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 939432 T3305 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 939437 T3305 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 939440 T3305 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 939442 T3305 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 939444 T3305 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 939448 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 939449 T3318 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@58a61f33 name:ZooKeeperConnection Watcher:127.0.0.1:40069/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 939450 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 939452 T3305 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 939454 T3305 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 939458 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 939460 T3320 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@78905076 name:ZooKeeperConnection Watcher:127.0.0.1:40069/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 939460 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 939461 T3305 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 939464 T3305 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 939470 T3305 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:40069_solr
   [junit4]   2> 939470 T3305 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 939472 T3305 oasc.Overseer.start Overseer (id=92069109050900484-127.0.0.1:40069_solr-n_0000000000) starting
   [junit4]   2> 939474 T3305 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 939477 T3305 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 939480 T3305 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 939483 T3305 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 939485 T3305 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 939488 T3305 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 939493 T3322 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 939495 T3321 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 939497 T3321 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 939498 T3321 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 939498 T3321 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
   [junit4]   2> 939498 T3321 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 939500 T3320 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 939501 T3318 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 939501 T3312 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 939996 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 940001 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 940003 T3320 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 940003 T3305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 940004 T3321 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 940005 T3307 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x147185debc40003, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 940005 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 940006 T3324 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@64e964c1 name:ZooKeeperConnection Watcher:127.0.0.1:40069/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 940006 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 940007 T3320 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 940007 T3305 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 940008 T3305 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 940010 T3320 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 940010 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 940011 T3321 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 940011 T3321 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 940012 T3320 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 940013 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 940015 T3320 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 940017 T3320 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 940118 T3312 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 940118 T3324 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 940119 T3305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 940121 T3305 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:40069 40069
   [junit4]   2> 940661 T3306 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:40069 40069
   [junit4]   2> 940662 T3305 oas.SolrTestCaseJ4.tearDown ###Ending testDoubleAssignment
   [junit4]   2> 940663 T3321 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:40069_solr
   [junit4]   2> 940664 T3325 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
   [junit4]   2> 
   [junit4]   2> 940667 T3305 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignment
   [junit4]   2> 940667 T3305 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 940668 T3326 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 940768 T3305 oasc.ZkTestServer.run start zk server on port:36765
   [junit4]   2> 940769 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 940771 T3332 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@32472cc8 name:ZooKeeperConnection Watcher:127.0.0.1:36765 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 940771 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 940773 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 940774 T3334 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3e820d75 name:ZooKeeperConnection Watcher:127.0.0.1:36765 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 940774 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 940775 T3305 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 940777 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 940778 T3336 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d83bcad name:ZooKeeperConnection Watcher:127.0.0.1:36765/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 940778 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 940779 T3305 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 940780 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 940781 T3338 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@50cec160 name:ZooKeeperConnection Watcher:127.0.0.1:36765/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 940781 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 940782 T3305 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 940783 T3305 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 940786 T3305 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:36765_solr
   [junit4]   2> 940786 T3305 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 940787 T3305 oasc.Overseer.start Overseer (id=92069109139243011-127.0.0.1:36765_solr-n_0000000000) starting
   [junit4]   2> 940788 T3305 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 940790 T3305 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 940792 T3305 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 940793 T3305 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 940795 T3305 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 940797 T3305 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 940799 T3340 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 940800 T3305 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 940800 T3339 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 940801 T3305 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 940802 T3305 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 940804 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 940805 T3342 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@799108df name:ZooKeeperConnection Watcher:127.0.0.1:36765/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 940805 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 940806 T3305 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 940807 T3305 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1
   [junit4]   2> 940809 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 940810 T3339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 940810 T3339 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 940810 T3339 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 940811 T3339 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 940812 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 940812 T3342 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 940812 T3336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 941309 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 941317 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 941320 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 941321 T3339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 941321 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 941323 T3339 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core2",
   [junit4]   2> 	  "core_node_name":"node2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 941323 T3339 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 941323 T3339 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 941324 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 941426 T3336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 941426 T3342 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 941821 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 941824 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 941827 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 941827 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 941827 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 941827 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 941828 T3339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 941829 T3339 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core3",
   [junit4]   2> 	  "core_node_name":"node3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 941830 T3339 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 941830 T3339 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 941831 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 941933 T3342 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 941933 T3336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 942328 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 942333 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 942335 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 942335 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 942336 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 942336 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 942336 T3339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 942338 T3339 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core4",
   [junit4]   2> 	  "core_node_name":"node4",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 942338 T3339 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 942338 T3339 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 942340 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 942441 T3342 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 942441 T3336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 942836 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 942842 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 942842 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 942842 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 942843 T3339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 942843 T3339 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core5",
   [junit4]   2> 	  "core_node_name":"node5",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 942844 T3339 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 942844 T3339 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 942845 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 942947 T3342 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 942947 T3336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 943342 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 943349 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 943349 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 943349 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 943349 T3339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 943350 T3339 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core6",
   [junit4]   2> 	  "core_node_name":"node6",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 943350 T3339 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 943350 T3339 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 943351 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 943453 T3342 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 943453 T3336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 943849 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 943860 T3305 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:36765 36765
   [junit4]   2> 944414 T3326 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:36765 36765
   [junit4]   2> 944416 T3305 oas.SolrTestCaseJ4.tearDown ###Ending testShardAssignment
   [junit4]   2> 944416 T3339 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:36765_solr
   [junit4]   2> 944418 T3343 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
   [junit4]   2> 
   [junit4]   2> 944424 T3305 oas.SolrTestCaseJ4.setUp ###Starting testShardLeaderChange
   [junit4]   2> 944425 T3305 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 944426 T3344 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 944526 T3305 oasc.ZkTestServer.run start zk server on port:44512
   [junit4]   2> 944527 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 944529 T3350 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f51554e name:ZooKeeperConnection Watcher:127.0.0.1:44512/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 944530 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 944530 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 944532 T3352 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6fc4a241 name:ZooKeeperConnection Watcher:127.0.0.1:44512 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 944532 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 944534 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 944535 T3354 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@41c9d817 name:ZooKeeperConnection Watcher:127.0.0.1:44512 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 944535 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 944536 T3305 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 944537 T3305 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 944540 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 944540 T3355 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 944541 T3358 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@556ad759 name:ZooKeeperConnection Watcher:127.0.0.1:44512/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 944541 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 944541 T3359 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@59603fde name:ZooKeeperConnection Watcher:127.0.0.1:44512/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 944542 T3355 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 944543 T3305 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 944543 T3355 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 944545 T3305 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 944545 T3355 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 944547 T3305 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 944549 T3355 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:44512_solr
   [junit4]   2> 944549 T3355 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 944549 T3305 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 944550 T3355 oasc.Overseer.start Overseer (id=92069109385527300-127.0.0.1:44512_solr-n_0000000000) starting
   [junit4]   2> 944552 T3355 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 944552 T3305 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 944555 T3355 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 944559 T3355 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 944563 T3355 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 944566 T3355 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 944570 T3355 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 944575 T3355 oasc.OverseerTest$OverseerRestarter.run Killing overseer.
   [junit4]   2> 944576 T3361 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 944577 T3355 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 944579 T3363 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@38767d5 name:ZooKeeperConnection Watcher:127.0.0.1:44512/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 944579 T3355 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 944580 T3361 oasc.OverseerCollectionProcessor.amILeader According to ZK I (id=92069109385527300-127.0.0.1:44512_solr-n_0000000000) am no longer a leader.
   [junit4]   2> 944581 T3361 oasc.SolrException.log ERROR :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/collection-queue-work
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 		at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 		at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 		at org.apache.solr.cloud.DistributedQueue.orderedChildren(DistributedQueue.java:99)
   [junit4]   2> 		at org.apache.solr.cloud.DistributedQueue.getTailId(DistributedQueue.java:483)
   [junit4]   2> 		at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:241)
   [junit4]   2> 		at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	
   [junit4]   2> 944581 T3361 oasc.OverseerCollectionProcessor.run ERROR Unable to prioritize overseer  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /roles.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:226)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:223)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:223)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionProcessor.prioritizeOverseerNodes(OverseerCollectionProcessor.java:412)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:256)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 944584 T3355 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:44512_solr
   [junit4]   2> 944584 T3355 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 944587 T3355 oasc.Overseer.start Overseer (id=92069109385527301-127.0.0.1:44512_solr-n_0000000001) starting
   [junit4]   2> 944600 T3365 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 944601 T3364 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 944602 T3364 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 944603 T3364 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 944603 T3364 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
   [junit4]   2> 944603 T3364 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 944605 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 944606 T3358 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 945055 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 945060 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 945062 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 945063 T3364 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 945063 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 945063 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 945065 T3364 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 945065 T3364 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 945067 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 945067 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 945068 T3367 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4326cfea name:ZooKeeperConnection Watcher:127.0.0.1:44512/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 945068 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 945069 T3305 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 945070 T3305 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
   [junit4]   2> 945073 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 945073 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 945073 T3364 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 945073 T3364 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 945075 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 945076 T3305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 945176 T3358 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 945177 T3367 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 945277 T3305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 945279 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 945280 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 945280 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 945281 T3364 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 945281 T3364 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"node2",
   [junit4]   2> 	  "core":"core4",
   [junit4]   2> 	  "core_node_name":"node2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node2/solr/"}
   [junit4]   2> 945281 T3364 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
   [junit4]   2> 945282 T3364 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 945283 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 945359 T3342 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 945359 T3342 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 945360 T3342 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 945385 T3358 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 945385 T3367 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 945780 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 945785 T3305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 945785 T3367 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 945787 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 945787 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 945788 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 945788 T3364 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 945789 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 945900 T3367 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 945986 T3305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 945988 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 945990 T3369 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@57a8f785 name:ZooKeeperConnection Watcher:127.0.0.1:44512/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 945990 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 945991 T3305 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 945993 T3305 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 945996 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 945996 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 945996 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 945997 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 945998 T3364 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 945998 T3364 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 945998 T3364 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 945999 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946001 T3369 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 946001 T3355 oasc.OverseerTest$OverseerRestarter.run Killing overseer.
   [junit4]   2> 946002 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946002 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 946003 T3355 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 946004 T3371 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4ccba4f1 name:ZooKeeperConnection Watcher:127.0.0.1:44512/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 946004 T3355 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 946004 T3364 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:44512_solr
   [junit4]   2> 946007 T3372 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
   [junit4]   2> 
   [junit4]   2> 946009 T3355 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:44512_solr
   [junit4]   2> 946009 T3355 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 946009 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 946011 T3374 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f921be7 name:ZooKeeperConnection Watcher:127.0.0.1:44512/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 946011 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 946011 T3355 oasc.Overseer.start Overseer (id=92069109385527304-127.0.0.1:44512_solr-n_0000000002) starting
   [junit4]   2> 946012 T3305 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 946013 T3305 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
   [junit4]   2> 946015 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 946017 T3376 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 946018 T3375 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 946018 T3305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 946019 T3375 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue.
   [junit4]   2> 946019 T3375 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 946019 T3375 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 946020 T3374 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 946020 T3369 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 946020 T3375 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 946021 T3375 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 946022 T3375 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 946022 T3375 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 946023 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946025 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946025 T3375 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 946025 T3375 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 946027 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946129 T3369 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 946129 T3374 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 946220 T3305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 946222 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946222 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946222 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946222 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 946223 T3375 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 946224 T3375 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"node2",
   [junit4]   2> 	  "core":"core4",
   [junit4]   2> 	  "core_node_name":"node2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node2/solr/"}
   [junit4]   2> 946224 T3375 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 946225 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946227 T3305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 946227 T3374 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 946232 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946234 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946337 T3374 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 946429 T3305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 946431 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 946433 T3378 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a677b49 name:ZooKeeperConnection Watcher:127.0.0.1:44512/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 946433 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 946435 T3305 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 946439 T3305 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 946444 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946444 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 946445 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946445 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946447 T3375 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 946448 T3375 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 946448 T3375 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 946451 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946454 T3378 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 946454 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946454 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 946455 T3375 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 946455 T3375 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 946456 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946458 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946458 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 946459 T3380 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@45eb95a2 name:ZooKeeperConnection Watcher:127.0.0.1:44512/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 946459 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 946460 T3305 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 946461 T3305 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
   [junit4]   2> 946464 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946464 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 946464 T3375 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 946465 T3375 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 946466 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946467 T3305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 946517 T3355 oasc.OverseerTest$OverseerRestarter.run Killing overseer.
   [junit4]   2> 946519 T3355 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 946520 T3382 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2eb03948 name:ZooKeeperConnection Watcher:127.0.0.1:44512/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 946521 T3355 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 946521 T3375 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:44512_solr
   [junit4]   2> 946522 T3383 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
   [junit4]   2> 
   [junit4]   2> 946525 T3355 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:44512_solr
   [junit4]   2> 946526 T3355 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 946528 T3355 oasc.Overseer.start Overseer (id=92069109385527308-127.0.0.1:44512_solr-n_0000000003) starting
   [junit4]   2> 946536 T3385 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 946538 T3384 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 946538 T3384 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue.
   [junit4]   2> 946539 T3384 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 946539 T3384 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 946540 T3380 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 946540 T3378 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 946543 T3384 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 946544 T3384 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 946544 T3380 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 946544 T3378 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 946548 T3378 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 946548 T3380 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 946551 T3384 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 946551 T3384 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 946552 T3380 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 946552 T3378 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 946554 T3384 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 946668 T3305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 946670 T3382 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946670 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 946672 T3384 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 946673 T3384 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"node2",
   [junit4]   2> 	  "core":"core4",
   [junit4]   2> 	  "core_node_name":"node2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node2/solr/"}
   [junit4]   2> 946673 T3384 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 946675 T3382 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946676 T3305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 946676 T3380 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 946678 T3382 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946680 T3382 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946782 T3380 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 946877 T3305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 946880 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 946881 T3387 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ca7fce name:ZooKeeperConnection Watcher:127.0.0.1:44512/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 946882 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 946883 T3305 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 946887 T3305 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 946892 T3382 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946892 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 946892 T3382 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946893 T3382 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946894 T3384 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 946894 T3384 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 946895 T3384 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 946896 T3382 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946898 T3387 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 946899 T3382 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946899 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 946899 T3384 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 946899 T3384 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 946901 T3382 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946902 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 946902 T3382 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946903 T3389 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3b53f3c1 name:ZooKeeperConnection Watcher:127.0.0.1:44512/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 946903 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 946904 T3305 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 946905 T3305 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
   [junit4]   2> 946908 T3382 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946908 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 946909 T3384 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 946909 T3384 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 946912 T3382 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946915 T3305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 946936 T3355 oasc.OverseerTest$OverseerRestarter.run Killing overseer.
   [junit4]   2> 946939 T3355 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 946941 T3391 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@33e81c41 name:ZooKeeperConnection Watcher:127.0.0.1:44512/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 946941 T3355 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 946942 T3384 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:44512_solr
   [junit4]   2> 946944 T3392 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apa

[...truncated too long message...]

ctions/collection1/leaders/shard1
   [junit4]   2> 971008 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 972011 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 973013 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 974015 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 975018 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 976020 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 977022 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 978024 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 979027 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 980030 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 981032 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 981036 T3305 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:52309 52309
   [junit4]   2> 981828 T3852 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:52309 52309
   [junit4]   2> 981829 T3305 oas.SolrTestCaseJ4.tearDown ###Ending testOverseerFailure
   [junit4]   2> 981830 T3872 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:52309_solr
   [junit4]   2> 981830 T3874 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=OverseerTest -Dtests.method=testOverseerFailure -Dtests.seed=B9E2DD92FCD61EDC -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=el -Dtests.timezone=GB -Dtests.file.encoding=UTF-8
   [junit4] ERROR   18.7s J0 | OverseerTest.testOverseerFailure <<<
   [junit4]    > Throwable #1: org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([B9E2DD92FCD61EDC:BDEA5261EE73F1FD]:0)
   [junit4]    > 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:143)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:155)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:314)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
   [junit4]    > 	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
   [junit4]    > 	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:660)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]    > 	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
   [junit4]    > 	at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:135)
   [junit4]    > 	at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31)
   [junit4]    > 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:130)
   [junit4]    > 	... 47 more
   [junit4]   2> 984839 T3305 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.OverseerTest-B9E2DD92FCD61EDC-001
   [junit4]   2> NOTE: test params are: codec=Lucene49: {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=el, timezone=GB
   [junit4]   2> NOTE: Linux 3.8.0-42-generic amd64/Oracle Corporation 1.7.0_60 (64-bit)/cpus=8,threads=1,free=107387592,total=390070272
   [junit4]   2> NOTE: All tests run in this JVM: [TestStressReorder, BlockCacheTest, ShardRoutingTest, DistributedSpellCheckComponentTest, AnalyticsMergeStrategyTest, TestCoreDiscovery, HdfsWriteToMultipleCollectionsTest, OverseerCollectionProcessorTest, TestFastWriter, ExternalFileFieldSortTest, TestDFRSimilarityFactory, MBeansHandlerTest, HighlighterTest, TestManagedSchema, TestFieldTypeResource, TestManagedResource, SpellingQueryConverterTest, SpellCheckComponentTest, ResponseHeaderTest, TimeZoneUtilsTest, AnalysisErrorHandlingTest, TestFastOutputStream, ReplicationFactorTest, TestFuzzyAnalyzedSuggestions, TestManagedStopFilterFactory, NoCacheHeaderTest, FastVectorHighlighterTest, SimpleFacetsTest, TestStandardQParsers, TestStressVersions, TestSearchPerf, TestPartialUpdateDeduplication, BadComponentTest, XsltUpdateRequestHandlerTest, TestAnalyzedSuggestions, TestDocBasedVersionConstraints, SuggesterWFSTTest, TestChildDocTransformer, TestSort, TestInfoStreamLogging, HttpPartitionTest, DateRangeFieldTest, RAMDirectoryFactoryTest, PathHierarchyTokenizerFactoryTest, CurrencyFieldOpenExchangeTest, ChaosMonkeyNothingIsSafeTest, AliasIntegrationTest, TestJmxIntegration, OverseerStatusTest, TestSchemaSimilarityResource, TestModifyConfFiles, ShowFileRequestHandlerTest, ResponseLogComponentTest, SpellPossibilityIteratorTest, TestCloudManagedSchema, TestCodecSupport, ZkNodePropsTest, TestDistributedSearch, TestComplexPhraseQParserPlugin, NotRequiredUniqueKeyTest, BadCopyFieldTest, BasicFunctionalityTest, TestSolrIndexConfig, HdfsSyncSliceTest, DocumentAnalysisRequestHandlerTest, TestOrdValues, AbstractAnalyticsStatsTest, TestSolr4Spatial, DisMaxRequestHandlerTest, JsonLoaderTest, DistribDocExpirationUpdateProcessorTest, HighlighterMaxOffsetTest, ZkControllerTest, IndexSchemaTest, DocValuesMultiTest, ParsingFieldUpdateProcessorsTest, TestLMDirichletSimilarityFactory, TestLeaderElectionZkExpiry, TestRemoteStreaming, BasicDistributedZkTest, RequiredFieldsTest, SuggesterTest, CoreAdminHandlerTest, FileUtilsTest, ExpressionTest, TestSolrDeletionPolicy1, TestReplicationHandlerBackup, HdfsBasicDistributedZkTest, TestReRankQParserPlugin, PrimitiveFieldTypeTest, TestSchemaResource, TestSolrXml, TestCustomSort, RecoveryZkTest, DateFieldTest, TestDynamicFieldResource, TestSolrJ, FunctionTest, WordBreakSolrSpellCheckerTest, StatelessScriptUpdateProcessorFactoryTest, SyncSliceTest, OverseerTest]
   [junit4] Completed on J0 in 45.55s, 9 tests, 1 error <<< FAILURES!

[...truncated 619 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1298: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:922: There were test failures: 417 suites, 1708 tests, 1 error, 38 ignored (17 assumptions)

Total time: 67 minutes 28 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.7.0_60 -XX:-UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_05) - Build # 10761 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10761/
Java: 64bit/jdk1.8.0_05 -XX:-UseCompressedOops -XX:+UseParallelGC

2 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
shard2 is not consistent.  Got 220 from https://127.0.0.1:57716/rm_/o/collection1lastClient and got 165 from https://127.0.0.1:39035/rm_/o/collection1

Stack Trace:
java.lang.AssertionError: shard2 is not consistent.  Got 220 from https://127.0.0.1:57716/rm_/o/collection1lastClient and got 165 from https://127.0.0.1:39035/rm_/o/collection1
	at __randomizedtesting.SeedInfo.seed([AAFB06BE7262B15:8B493E7390794B29]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1132)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1111)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:150)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:863)
	at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.RollingRestartTest.testDistribSearch

Error Message:
Unable to restart (#1): CloudJettyRunner [url=https://127.0.0.1:54317/_vn/gc/collection1]

Stack Trace:
java.lang.AssertionError: Unable to restart (#1): CloudJettyRunner [url=https://127.0.0.1:54317/_vn/gc/collection1]
	at __randomizedtesting.SeedInfo.seed([AAFB06BE7262B15:8B493E7390794B29]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.RollingRestartTest.restartWithRolesTest(RollingRestartTest.java:102)
	at org.apache.solr.cloud.RollingRestartTest.doTest(RollingRestartTest.java:61)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:863)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11271 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/init-core-data-001
   [junit4]   2> 682739 T3421 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 682740 T3421 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /rm_/o
   [junit4]   2> 682744 T3421 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 682745 T3421 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 682745 T3422 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 682845 T3421 oasc.ZkTestServer.run start zk server on port:52362
   [junit4]   2> 682846 T3421 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 682848 T3428 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b0954aa name:ZooKeeperConnection Watcher:127.0.0.1:52362 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 682848 T3421 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 682849 T3421 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 682851 T3421 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 682852 T3430 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1eeccf84 name:ZooKeeperConnection Watcher:127.0.0.1:52362/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 682852 T3421 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 682852 T3421 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 682854 T3421 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 682855 T3421 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 682855 T3421 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 682856 T3421 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 682857 T3421 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 682858 T3421 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 682858 T3421 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 682859 T3421 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 682859 T3421 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 682860 T3421 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 682861 T3421 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 682862 T3421 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 682862 T3421 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 682863 T3421 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 682863 T3421 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 682864 T3421 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 682864 T3421 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 682865 T3421 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 682866 T3421 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 682866 T3421 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 682867 T3421 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 682868 T3421 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 682868 T3421 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 682869 T3421 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 682870 T3421 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 682871 T3421 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 682872 T3432 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12cb7afa name:ZooKeeperConnection Watcher:127.0.0.1:52362/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 682872 T3421 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 683092 T3421 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 683094 T3421 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 683097 T3421 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:58033
   [junit4]   2> 683098 T3421 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 683098 T3421 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 683099 T3421 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-002
   [junit4]   2> 683099 T3421 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-002/'
   [junit4]   2> 683110 T3421 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-002/solr.xml
   [junit4]   2> 683127 T3421 oasc.CoreContainer.<init> New CoreContainer 1285219428
   [junit4]   2> 683127 T3421 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-002/]
   [junit4]   2> 683128 T3421 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 683128 T3421 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 683128 T3421 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 683129 T3421 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 683129 T3421 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 683129 T3421 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 683129 T3421 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 683129 T3421 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 683130 T3421 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 683131 T3421 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 683131 T3421 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 683132 T3421 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 683132 T3421 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52362/solr
   [junit4]   2> 683132 T3421 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 683133 T3421 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 683134 T3443 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1739d36d name:ZooKeeperConnection Watcher:127.0.0.1:52362 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 683134 T3421 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 683136 T3421 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 683137 T3445 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c03e184 name:ZooKeeperConnection Watcher:127.0.0.1:52362/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 683137 T3421 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 683138 T3421 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 683139 T3421 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 683141 T3421 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 683142 T3421 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 683144 T3421 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 683145 T3421 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 683146 T3421 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58033_rm_%2Fo
   [junit4]   2> 683147 T3421 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58033_rm_%2Fo
   [junit4]   2> 683148 T3421 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 683149 T3421 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 683151 T3421 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:58033_rm_%2Fo
   [junit4]   2> 683151 T3421 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 683152 T3421 oasc.Overseer.start Overseer (id=92068493388873732-127.0.0.1:58033_rm_%2Fo-n_0000000000) starting
   [junit4]   2> 683153 T3421 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 683158 T3447 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 683158 T3421 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 683159 T3421 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 683160 T3421 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 683161 T3446 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 683163 T3448 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 683163 T3448 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 683163 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 683163 T3448 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 683164 T3446 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 683164 T3446 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:58033/rm_/o",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:58033_rm_%2Fo",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 683165 T3446 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 683165 T3446 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 683166 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 683166 T3445 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 684164 T3448 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 684164 T3448 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 684165 T3448 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 684165 T3448 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 684165 T3448 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 684165 T3448 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-002/collection1/'
   [junit4]   2> 684166 T3448 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 684166 T3448 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 684166 T3448 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 684182 T3448 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 684191 T3448 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 684192 T3448 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 684196 T3448 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 684288 T3448 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 684289 T3448 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 684290 T3448 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 684292 T3448 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 684302 T3448 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 684303 T3448 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 684312 T3448 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/control/data/
   [junit4]   2> 684312 T3448 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2dc8a1f9
   [junit4]   2> 684313 T3448 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/control/data
   [junit4]   2> 684313 T3448 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/control/data/index/
   [junit4]   2> 684313 T3448 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 684313 T3448 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/control/data/index
   [junit4]   2> 684314 T3448 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=16, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=59.9013671875, floorSegmentMB=1.90234375, forceMergeDeletesPctAllowed=19.856637667348704, segmentsPerTier=24.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 693751 T3448 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 693751 T3448 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 693754 T3448 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 693754 T3448 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 693754 T3448 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 693754 T3448 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 693754 T3448 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 693754 T3448 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 693755 T3448 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 693755 T3448 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 693755 T3448 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 693755 T3448 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 693755 T3448 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 693756 T3448 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 693756 T3448 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 693756 T3448 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 693756 T3448 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 693756 T3448 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 693757 T3448 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 693758 T3448 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 693759 T3448 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 693759 T3448 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 693759 T3448 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=36, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 693760 T3448 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 693760 T3448 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 693760 T3448 oass.SolrIndexSearcher.<init> Opening Searcher@3cb8561e[collection1] main
   [junit4]   2> 693761 T3448 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 693761 T3448 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 693761 T3448 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 693762 T3448 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 693762 T3448 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 693762 T3448 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 693762 T3448 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 693763 T3448 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 693764 T3448 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 693765 T3448 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 693765 T3448 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 693765 T3448 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 693768 T3449 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3cb8561e[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 693768 T3448 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 693769 T3452 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:58033/rm_/o collection:control_collection shard:shard1
   [junit4]   2> 693769 T3421 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 693769 T3452 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 693769 T3421 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 693772 T3421 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 693772 T3452 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 693772 T3454 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@56e26ba4 name:ZooKeeperConnection Watcher:127.0.0.1:52362/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 693773 T3421 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 693773 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 693773 T3452 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 693773 T3452 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 693773 T3421 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 693773 T3446 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C908 name=collection1 org.apache.solr.core.SolrCore@29733922 url=https://127.0.0.1:58033/rm_/o/collection1 node=127.0.0.1:58033_rm_%2Fo C908_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:58033/rm_/o, core=collection1, node_name=127.0.0.1:58033_rm_%2Fo}
   [junit4]   2> 693773 T3452 C908 P58033 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:58033/rm_/o/collection1/
   [junit4]   2> 693774 T3452 C908 P58033 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 693774 T3452 C908 P58033 oasc.SyncStrategy.syncToMe https://127.0.0.1:58033/rm_/o/collection1/ has no replicas
   [junit4]   2> 693774 T3452 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:58033/rm_/o/collection1/ shard1
   [junit4]   2> 693774 T3421 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 693775 T3452 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 693775 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 693775 T3445 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 693775 T3454 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 693777 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 693778 T3446 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 693779 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 693880 T3454 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 693880 T3445 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 693928 T3452 oasc.ZkController.register We are https://127.0.0.1:58033/rm_/o/collection1/ and leader is https://127.0.0.1:58033/rm_/o/collection1/
   [junit4]   2> 693928 T3452 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:58033/rm_/o
   [junit4]   2> 693929 T3452 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 693929 T3452 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 693929 T3452 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 693930 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 693930 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 693930 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 693930 T3452 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 693931 T3446 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 693931 T3446 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:58033/rm_/o",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:58033_rm_%2Fo",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 693934 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 694005 T3421 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 694006 T3421 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 694008 T3421 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 694009 T3421 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:60624
   [junit4]   2> 694011 T3421 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 694011 T3421 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 694011 T3421 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-003
   [junit4]   2> 694011 T3421 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-003/'
   [junit4]   2> 694021 T3421 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-003/solr.xml
   [junit4]   2> 694028 T3421 oasc.CoreContainer.<init> New CoreContainer 350194343
   [junit4]   2> 694028 T3421 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-003/]
   [junit4]   2> 694029 T3421 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 694029 T3421 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 694029 T3421 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 694029 T3421 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 694030 T3421 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 694030 T3421 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 694030 T3421 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 694030 T3421 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 694030 T3421 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 694032 T3421 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 694032 T3421 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 694033 T3421 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 694033 T3421 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52362/solr
   [junit4]   2> 694033 T3421 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 694034 T3421 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 694035 T3465 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e6c2ae1 name:ZooKeeperConnection Watcher:127.0.0.1:52362 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 694035 T3421 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 694035 T3445 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 694035 T3454 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 694036 T3421 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 694037 T3467 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6d29f73e name:ZooKeeperConnection Watcher:127.0.0.1:52362/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 694037 T3421 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 694040 T3421 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 695042 T3421 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60624_rm_%2Fo
   [junit4]   2> 695043 T3421 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60624_rm_%2Fo
   [junit4]   2> 695052 T3468 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 695052 T3468 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 695052 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 695052 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 695053 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 695052 T3468 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 695053 T3446 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 695054 T3446 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:60624/rm_/o",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:60624_rm_%2Fo",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 695054 T3446 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 695054 T3446 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 695055 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 695157 T3454 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 695157 T3445 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 695157 T3467 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 696053 T3468 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 696053 T3468 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 696054 T3468 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 696054 T3468 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 696055 T3468 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 696055 T3468 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-003/collection1/'
   [junit4]   2> 696056 T3468 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 696056 T3468 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 696056 T3468 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 696081 T3468 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 696090 T3468 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 696091 T3468 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 696095 T3468 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 696173 T3468 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 696174 T3468 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 696175 T3468 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 696177 T3468 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 696187 T3468 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 696187 T3468 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 696187 T3468 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-003/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/jetty1/
   [junit4]   2> 696187 T3468 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2dc8a1f9
   [junit4]   2> 696188 T3468 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/jetty1
   [junit4]   2> 696188 T3468 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/jetty1/index/
   [junit4]   2> 696189 T3468 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 696189 T3468 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/jetty1/index
   [junit4]   2> 696189 T3468 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=16, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=59.9013671875, floorSegmentMB=1.90234375, forceMergeDeletesPctAllowed=19.856637667348704, segmentsPerTier=24.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 696198 T3468 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 696198 T3468 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 696201 T3468 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 696201 T3468 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 696201 T3468 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 696202 T3468 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 696202 T3468 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 696202 T3468 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 696202 T3468 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 696202 T3468 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 696202 T3468 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 696203 T3468 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 696203 T3468 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 696203 T3468 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 696203 T3468 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 696203 T3468 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 696204 T3468 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 696204 T3468 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 696204 T3468 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 696206 T3468 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 696207 T3468 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 696207 T3468 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 696207 T3468 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=36, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 696208 T3468 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 696208 T3468 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 696208 T3468 oass.SolrIndexSearcher.<init> Opening Searcher@62920d54[collection1] main
   [junit4]   2> 696208 T3468 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 696209 T3468 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 696210 T3468 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 696210 T3468 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 696210 T3468 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 696210 T3468 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 696211 T3468 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 696211 T3468 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 696211 T3468 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 696211 T3468 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 696213 T3469 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@62920d54[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 696214 T3468 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 696215 T3472 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:60624/rm_/o collection:collection1 shard:shard2
   [junit4]   2> 696215 T3421 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 696215 T3421 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 696215 T3472 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 696218 T3472 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 696219 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 696219 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 696219 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 696220 T3472 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 696220 T3472 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 696220 T3446 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C909 name=collection1 org.apache.solr.core.SolrCore@5e375d0e url=https://127.0.0.1:60624/rm_/o/collection1 node=127.0.0.1:60624_rm_%2Fo C909_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:60624/rm_/o, core=collection1, node_name=127.0.0.1:60624_rm_%2Fo}
   [junit4]   2> 696220 T3472 C909 P60624 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:60624/rm_/o/collection1/
   [junit4]   2> 696220 T3472 C909 P60624 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 696221 T3472 C909 P60624 oasc.SyncStrategy.syncToMe https://127.0.0.1:60624/rm_/o/collection1/ has no replicas
   [junit4]   2> 696221 T3472 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:60624/rm_/o/collection1/ shard2
   [junit4]   2> 696221 T3472 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 696221 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 696225 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 696226 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 696327 T3454 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 696327 T3467 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 696327 T3445 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 696376 T3472 oasc.ZkController.register We are https://127.0.0.1:60624/rm_/o/collection1/ and leader is https://127.0.0.1:60624/rm_/o/collection1/
   [junit4]   2> 696376 T3472 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:60624/rm_/o
   [junit4]   2> 696376 T3472 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 696376 T3472 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 696376 T3472 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 696378 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 696378 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 696378 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 696378 T3472 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 696379 T3446 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 696379 T3446 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:60624/rm_/o",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:60624_rm_%2Fo",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 696380 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 696437 T3421 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 696438 T3421 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 696440 T3421 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 696441 T3421 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:42159
   [junit4]   2> 696443 T3421 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 696443 T3421 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 696443 T3421 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-004
   [junit4]   2> 696443 T3421 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-004/'
   [junit4]   2> 696453 T3421 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-004/solr.xml
   [junit4]   2> 696460 T3421 oasc.CoreContainer.<init> New CoreContainer 1136505569
   [junit4]   2> 696461 T3421 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-004/]
   [junit4]   2> 696461 T3421 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 696461 T3421 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 696461 T3421 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 696462 T3421 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 696462 T3421 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 696462 T3421 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 696462 T3421 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 696462 T3421 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 696463 T3421 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 696464 T3421 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 696464 T3421 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 696465 T3421 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 696465 T3421 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52362/solr
   [junit4]   2> 696465 T3421 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 696466 T3421 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 696467 T3483 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f29bc8c name:ZooKeeperConnection Watcher:127.0.0.1:52362 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 696467 T3421 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 696469 T3421 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 696469 T3485 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3e1acc85 name:ZooKeeperConnection Watcher:127.0.0.1:52362/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 696469 T3421 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 696473 T3421 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 696481 T3454 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 696481 T3445 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 696481 T3485 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 696481 T3467 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 697474 T3421 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42159_rm_%2Fo
   [junit4]   2> 697475 T3421 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42159_rm_%2Fo
   [junit4]   2> 697479 T3486 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 697480 T3486 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 697481 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 697481 T3486 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 697481 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 697481 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 697481 T3446 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 697482 T3446 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:42159/rm_/o",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:42159_rm_%2Fo",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 697482 T3446 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 697482 T3446 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 697483 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 697584 T3454 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 697584 T3445 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 697584 T3485 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 697584 T3467 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 698481 T3486 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 698482 T3486 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 698482 T3486 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 698482 T3486 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 698483 T3486 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 698483 T3486 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-004/collection1/'
   [junit4]   2> 698484 T3486 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 698484 T3486 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 698485 T3486 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 698507 T3486 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 698524 T3486 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 698525 T3486 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 698531 T3486 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 698613 T3486 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 698615 T3486 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 698615 T3486 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 698617 T3486 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 698629 T3486 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 698629 T3486 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 698629 T3486 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-004/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/jetty2/
   [junit4]   2> 698629 T3486 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2dc8a1f9
   [junit4]   2> 698630 T3486 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/jetty2
   [junit4]   2> 698630 T3486 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/jetty2/index/
   [junit4]   2> 698630 T3486 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 698630 T3486 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/jetty2/index
   [junit4]   2> 698631 T3486 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=16, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=59.9013671875, floorSegmentMB=1.90234375, forceMergeDeletesPctAllowed=19.856637667348704, segmentsPerTier=24.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 698645 T3486 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 698645 T3486 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 698648 T3486 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 698648 T3486 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 698648 T3486 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 698648 T3486 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 698648 T3486 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 698649 T3486 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 698649 T3486 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 698649 T3486 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 698649 T3486 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 698649 T3486 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 698650 T3486 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 698650 T3486 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 698650 T3486 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 698650 T3486 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 698650 T3486 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 698651 T3486 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 698651 T3486 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 698652 T3486 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 698653 T3486 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 698653 T3486 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 698654 T3486 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=36, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 698655 T3486 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 698655 T3486 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 698655 T3486 oass.SolrIndexSearcher.<init> Opening Searcher@6d7f5127[collection1] main
   [junit4]   2> 698655 T3486 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 698656 T3486 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 698656 T3486 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 698657 T3486 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 698657 T3486 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 698657 T3486 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 698658 T3486 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 698658 T3486 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 698658 T3486 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 698658 T3486 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 698660 T3487 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6d7f5127[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 698661 T3486 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 698662 T3490 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:42159/rm_/o collection:collection1 shard:shard3
   [junit4]   2> 698662 T3421 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 698662 T3490 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 698662 T3421 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 698665 T3490 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
   [junit4]   2> 698665 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 698666 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 698666 T3490 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 698666 T3490 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 698666 T3446 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 698666 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> ASYNC  NEW_CORE C910 name=collection1 org.apache.solr.core.SolrCore@51c3a3c9 url=https://127.0.0.1:42159/rm_/o/collection1 node=127.0.0.1:42159_rm_%2Fo C910_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:42159/rm_/o, core=collection1, node_name=127.0.0.1:42159_rm_%2Fo}
   [junit4]   2> 698666 T3490 C910 P42159 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:42159/rm_/o/collection1/
   [junit4]   2> 698667 T3490 C910 P42159 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 698667 T3490 C910 P42159 oasc.SyncStrategy.syncToMe https://127.0.0.1:42159/rm_/o/collection1/ has no replicas
   [junit4]   2> 698667 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 698667 T3490 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:42159/rm_/o/collection1/ shard3
   [junit4]   2> 698668 T3490 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 698669 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 698671 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 698772 T3454 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 698772 T3445 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 698772 T3485 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 698772 T3467 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 698820 T3490 oasc.ZkController.register We are https://127.0.0.1:42159/rm_/o/collection1/ and leader is https://127.0.0.1:42159/rm_/o/collection1/
   [junit4]   2> 698820 T3490 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:42159/rm_/o
   [junit4]   2> 698821 T3490 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 698821 T3490 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 698821 T3490 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 698822 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 698822 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 698822 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 698822 T3490 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 698823 T3446 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 698824 T3446 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:42159/rm_/o",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:42159_rm_%2Fo",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 698825 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 698887 T3421 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 698887 T3421 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 698890 T3421 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 698890 T3421 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:37470
   [junit4]   2> 698892 T3421 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 698892 T3421 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 698893 T3421 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-005
   [junit4]   2> 698893 T3421 oas

[...truncated too long message...]

losing IndexWriter
   [junit4]   2> 1083433 T3918 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 1083434 T3918 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1083435 T3918 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1083435 T3918 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-AAFB06BE7262B15-001/tempDir-001/jetty15 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-AAFB06BE7262B15-001/tempDir-001/jetty15;done=false>>]
   [junit4]   2> 1083435 T3918 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-AAFB06BE7262B15-001/tempDir-001/jetty15
   [junit4]   2> 1083435 T3918 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-AAFB06BE7262B15-001/tempDir-001/jetty15/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-AAFB06BE7262B15-001/tempDir-001/jetty15/index;done=false>>]
   [junit4]   2> 1083436 T3918 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-AAFB06BE7262B15-001/tempDir-001/jetty15/index
   [junit4]   2> 1083436 T3918 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 1083437 T3918 oasc.SolrException.log ERROR There was a problem trying to register as the leader:org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:143)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:298)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 	Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 		at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:209)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:206)
   [junit4]   2> 		at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:206)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:417)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:135)
   [junit4]   2> 		at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:130)
   [junit4]   2> 		... 7 more
   [junit4]   2> 	
   [junit4]   2> 1083437 T3918 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1083438 T3918 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-AAFB06BE7262B15-001
   [junit4]   2> NOTE: test params are: codec=Lucene45, sim=DefaultSimilarity, locale=nl_NL, timezone=US/Samoa
   [junit4]   2> NOTE: Linux 3.8.0-42-generic amd64/Oracle Corporation 1.8.0_05 (64-bit)/cpus=8,threads=1,free=142840936,total=317194240
   [junit4]   2> NOTE: All tests run in this JVM: [TestDistribDocBasedVersion, DirectSolrSpellCheckerTest, TestFoldingMultitermQuery, TestBlendedInfixSuggestions, TestWordDelimiterFilterFactory, XmlUpdateRequestHandlerTest, TestCodecSupport, BasicDistributedZkTest, SolrCoreCheckLockOnStartupTest, PluginInfoTest, JsonLoaderTest, BasicFunctionalityTest, TestSolr4Spatial, TestTrie, BadIndexSchemaTest, PingRequestHandlerTest, TestComponentsName, TestHighFrequencyDictionaryFactory, TestCollectionAPI, RegexBoostProcessorTest, ReturnFieldsTest, StressHdfsTest, PreAnalyzedUpdateProcessorTest, TestNonNRTOpen, SpellPossibilityIteratorTest, FieldMutatingUpdateProcessorTest, XsltUpdateRequestHandlerTest, SuggesterWFSTTest, SignatureUpdateProcessorFactoryTest, TestFieldTypeCollectionResource, ShowFileRequestHandlerTest, FieldFacetTest, ModifyConfFileTest, SimpleFacetsTest, SolrIndexSplitterTest, TestDynamicFieldCollectionResource, ShardRoutingTest, TestDocBasedVersionConstraints, OpenExchangeRatesOrgProviderTest, TestUniqueKeyFieldResource, SSLMigrationTest, UnloadDistributedZkTest, SuggestComponentTest, TestFastLRUCache, HdfsWriteToMultipleCollectionsTest, RankQueryTest, HdfsRecoveryZkTest, HdfsCollectionsAPIDistributedZkTest, HdfsBasicDistributedZk2Test, ExternalFileFieldSortTest, TestSchemaResource, PathHierarchyTokenizerFactoryTest, HighlighterMaxOffsetTest, JSONWriterTest, DocValuesMissingTest, ChaosMonkeyNothingIsSafeTest, TestStressRecovery, HttpPartitionTest, StandardRequestHandlerTest, SyncSliceTest, TestHighlightDedupGrouping, BadComponentTest, TestConfigSets, IndexSchemaTest, ZkNodePropsTest, TestManagedSynonymFilterFactory, HdfsDirectoryTest, FastVectorHighlighterTest, CustomCollectionTest, ConnectionManagerTest, SpellCheckCollatorTest, TestSolrDeletionPolicy2, MigrateRouteKeyTest, TestFastWriter, DocValuesMultiTest, TestDocSet, ZkCLITest, ZkControllerTest, TestMaxScoreQueryParser, TestShardHandlerFactory, TestManagedSchema, ResponseLogComponentTest, SolrCmdDistributorTest, TestReload, TestAddFieldRealTimeGet, ShardSplitTest, FieldFacetExtrasTest, TestStressReorder, TestCloudInspectUtil, CurrencyFieldOpenExchangeTest, RollingRestartTest]
   [junit4] Completed on J0 in 98.71s, 1 test, 1 failure <<< FAILURES!

[...truncated 638 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1298: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:922: There were test failures: 417 suites, 1708 tests, 2 failures, 38 ignored (17 assumptions)

Total time: 62 minutes 8 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_05 -XX:-UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



Re: [JENKINS] Lucene-Solr-trunk-Linux (32bit/ibm-j9-jdk7) - Build # 10760 - Still Failing!

Posted by "david.w.smiley@gmail.com" <da...@gmail.com>.
I’m on it; this’ll get fixed momentarily.  Some co-related JIRA issues; one
got committed without the other.

~ David Smiley
Freelance Apache Lucene/Solr Search Consultant/Developer
http://www.linkedin.com/in/davidwsmiley


On Tue, Jul 8, 2014 at 2:54 PM, Policeman Jenkins Server <
jenkins@thetaphi.de> wrote:

> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10760/
> Java: 32bit/ibm-j9-jdk7
> -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
>
> 2 tests failed.
> FAILED:  org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations
> {#5 seed=[CDF4E70BC4C20419:69B5EDAA7D26DA74]}
>
> Error Message:
> [Contains] Shouldn't match
> I#5:Rect(minX=-150.0,maxX=-80.0,minY=-90.0,maxY=-90.0)
> Q:Rect(minX=-110.0,maxX=-100.0,minY=-90.0,maxY=-90.0)
>
> Stack Trace:
> java.lang.AssertionError: [Contains] Shouldn't match
> I#5:Rect(minX=-150.0,maxX=-80.0,minY=-90.0,maxY=-90.0)
> Q:Rect(minX=-110.0,maxX=-100.0,minY=-90.0,maxY=-90.0)
>         at
> __randomizedtesting.SeedInfo.seed([CDF4E70BC4C20419:69B5EDAA7D26DA74]:0)
>         at org.junit.Assert.fail(Assert.java:93)
>         at
> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
>         at
> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
>         at
> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:62)
>         at
> org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations(TestBBoxStrategy.java:97)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
>         at java.lang.reflect.Method.invoke(Method.java:619)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>         at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>         at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>         at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>         at java.lang.Thread.run(Thread.java:853)
>
>
> FAILED:  org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations
> {#7 seed=[CDF4E70BC4C20419:CBD813564C025EDC]}
>
> Error Message:
> [Contains] Shouldn't match
> I#0:Rect(minX=-150.0,maxX=-50.0,minY=-70.0,maxY=-70.0)
> Q:Rect(minX=-150.0,maxX=-130.0,minY=-70.0,maxY=-70.0)
>
> Stack Trace:
> java.lang.AssertionError: [Contains] Shouldn't match
> I#0:Rect(minX=-150.0,maxX=-50.0,minY=-70.0,maxY=-70.0)
> Q:Rect(minX=-150.0,maxX=-130.0,minY=-70.0,maxY=-70.0)
>         at
> __randomizedtesting.SeedInfo.seed([CDF4E70BC4C20419:CBD813564C025EDC]:0)
>         at org.junit.Assert.fail(Assert.java:93)
>         at
> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
>         at
> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
>         at
> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:62)
>         at
> org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations(TestBBoxStrategy.java:97)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
>         at java.lang.reflect.Method.invoke(Method.java:619)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>         at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>         at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>         at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>         at java.lang.Thread.run(Thread.java:853)
>
>
>
>
> Build Log:
> [...truncated 9432 lines...]
>    [junit4] Suite: org.apache.lucene.spatial.bbox.TestBBoxStrategy
>    [junit4]   2> ??? 09, 2014 2:53:26 ?
> org.apache.lucene.spatial.StrategyTestCase executeQueries
>    [junit4]   2> INFO: testing queried for strategy BBoxStrategy
> field:bbox ctx=SpatialContext.GEO
>    [junit4]   2> ??? 09, 2014 2:53:26 ?
> org.apache.lucene.spatial.StrategyTestCase executeQueries
>    [junit4]   2> INFO: testing queried for strategy BBoxStrategy
> field:bbox ctx=SpatialContext.GEO
>    [junit4]   2> NOTE: reproduce with: ant test
>  -Dtestcase=TestBBoxStrategy -Dtests.method=testOperations
> -Dtests.seed=CDF4E70BC4C20419 -Dtests.multiplier=3 -Dtests.slow=true
> -Dtests.locale=ar_KW -Dtests.timezone=Etc/GMT-8
> -Dtests.file.encoding=US-ASCII
>    [junit4] FAILURE 0.22s J1 | TestBBoxStrategy.testOperations {#5
> seed=[CDF4E70BC4C20419:69B5EDAA7D26DA74]} <<<
>    [junit4]    > Throwable #1: java.lang.AssertionError: [Contains]
> Shouldn't match I#5:Rect(minX=-150.0,maxX=-80.0,minY=-90.0,maxY=-90.0)
> Q:Rect(minX=-110.0,maxX=-100.0,minY=-90.0,maxY=-90.0)
>    [junit4]    >        at
> __randomizedtesting.SeedInfo.seed([CDF4E70BC4C20419:69B5EDAA7D26DA74]:0)
>    [junit4]    >        at
> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
>    [junit4]    >        at
> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
>    [junit4]    >        at
> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:62)
>    [junit4]    >        at
> org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations(TestBBoxStrategy.java:97)
>    [junit4]    >        at java.lang.Thread.run(Thread.java:853)
>    [junit4]   2> NOTE: reproduce with: ant test
>  -Dtestcase=TestBBoxStrategy -Dtests.method=testOperations
> -Dtests.seed=CDF4E70BC4C20419 -Dtests.multiplier=3 -Dtests.slow=true
> -Dtests.locale=ar_KW -Dtests.timezone=Etc/GMT-8
> -Dtests.file.encoding=US-ASCII
>    [junit4] FAILURE 0.19s J1 | TestBBoxStrategy.testOperations {#7
> seed=[CDF4E70BC4C20419:CBD813564C025EDC]} <<<
>    [junit4]    > Throwable #1: java.lang.AssertionError: [Contains]
> Shouldn't match I#0:Rect(minX=-150.0,maxX=-50.0,minY=-70.0,maxY=-70.0)
> Q:Rect(minX=-150.0,maxX=-130.0,minY=-70.0,maxY=-70.0)
>    [junit4]    >        at
> __randomizedtesting.SeedInfo.seed([CDF4E70BC4C20419:CBD813564C025EDC]:0)
>    [junit4]    >        at
> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
>    [junit4]    >        at
> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
>    [junit4]    >        at
> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:62)
>    [junit4]    >        at
> org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations(TestBBoxStrategy.java:97)
>    [junit4]    >        at java.lang.Thread.run(Thread.java:853)
>    [junit4]   2> ??? 09, 2014 2:53:35 ?
> org.apache.lucene.spatial.StrategyTestCase executeQueries
>    [junit4]   2> INFO: testing queried for strategy BBoxStrategy
> field:bbox ctx=SpatialContext.GEO
>    [junit4]   2> NOTE: leaving temporary files on disk at:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build/spatial/test/J1/./temp/lucene.spatial.bbox.TestBBoxStrategy-CDF4E70BC4C20419-001
>    [junit4]   2> NOTE: test params are: codec=Lucene40,
> sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {bbox__minY=DFR
> I(n)B3(800.0), bbox__maxX=IB SPL-D1, bbox__maxY=IB LL-D1, bbox__xdl=DFR
> GZ(0.3), bbox__minX=DFR G1}, locale=ar_KW, timezone=Etc/GMT-8
>    [junit4]   2> NOTE: Linux 3.8.0-42-generic x86/IBM Corporation 1.7.0
> (32-bit)/cpus=8,threads=1,free=24357064,total=56819712
>    [junit4]   2> NOTE: All tests run in this JVM: [JtsPolygonTest,
> TestTermQueryPrefixGridStrategy, TestPointVectorStrategy,
> DistanceStrategyTest, SpatialPrefixTreeTest, TestBBoxStrategy]
>    [junit4] Completed on J1 in 10.26s, 30 tests, 2 failures, 1 skipped <<<
> FAILURES!
>
> [...truncated 20 lines...]
> BUILD FAILED
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:460: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build.xml:544:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:2019:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/module-build.xml:60:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1298:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:922:
> There were test failures: 16 suites, 143 tests, 2 failures, 13 ignored (2
> assumptions)
>
> Total time: 39 minutes 55 seconds
> Build step 'Invoke Ant' marked build as failure
> [description-setter] Description set: Java: 32bit/ibm-j9-jdk7
> -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure - Any
> Sending email for trigger: Failure - Any
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>

[JENKINS] Lucene-Solr-trunk-Linux (32bit/ibm-j9-jdk7) - Build # 10760 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10760/
Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}

2 tests failed.
FAILED:  org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations {#5 seed=[CDF4E70BC4C20419:69B5EDAA7D26DA74]}

Error Message:
[Contains] Shouldn't match I#5:Rect(minX=-150.0,maxX=-80.0,minY=-90.0,maxY=-90.0) Q:Rect(minX=-110.0,maxX=-100.0,minY=-90.0,maxY=-90.0)

Stack Trace:
java.lang.AssertionError: [Contains] Shouldn't match I#5:Rect(minX=-150.0,maxX=-80.0,minY=-90.0,maxY=-90.0) Q:Rect(minX=-110.0,maxX=-100.0,minY=-90.0,maxY=-90.0)
	at __randomizedtesting.SeedInfo.seed([CDF4E70BC4C20419:69B5EDAA7D26DA74]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:62)
	at org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations(TestBBoxStrategy.java:97)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:619)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:853)


FAILED:  org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations {#7 seed=[CDF4E70BC4C20419:CBD813564C025EDC]}

Error Message:
[Contains] Shouldn't match I#0:Rect(minX=-150.0,maxX=-50.0,minY=-70.0,maxY=-70.0) Q:Rect(minX=-150.0,maxX=-130.0,minY=-70.0,maxY=-70.0)

Stack Trace:
java.lang.AssertionError: [Contains] Shouldn't match I#0:Rect(minX=-150.0,maxX=-50.0,minY=-70.0,maxY=-70.0) Q:Rect(minX=-150.0,maxX=-130.0,minY=-70.0,maxY=-70.0)
	at __randomizedtesting.SeedInfo.seed([CDF4E70BC4C20419:CBD813564C025EDC]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:62)
	at org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations(TestBBoxStrategy.java:97)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:619)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:853)




Build Log:
[...truncated 9432 lines...]
   [junit4] Suite: org.apache.lucene.spatial.bbox.TestBBoxStrategy
   [junit4]   2> ??? 09, 2014 2:53:26 ? org.apache.lucene.spatial.StrategyTestCase executeQueries
   [junit4]   2> INFO: testing queried for strategy BBoxStrategy field:bbox ctx=SpatialContext.GEO
   [junit4]   2> ??? 09, 2014 2:53:26 ? org.apache.lucene.spatial.StrategyTestCase executeQueries
   [junit4]   2> INFO: testing queried for strategy BBoxStrategy field:bbox ctx=SpatialContext.GEO
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestBBoxStrategy -Dtests.method=testOperations -Dtests.seed=CDF4E70BC4C20419 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_KW -Dtests.timezone=Etc/GMT-8 -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 0.22s J1 | TestBBoxStrategy.testOperations {#5 seed=[CDF4E70BC4C20419:69B5EDAA7D26DA74]} <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: [Contains] Shouldn't match I#5:Rect(minX=-150.0,maxX=-80.0,minY=-90.0,maxY=-90.0) Q:Rect(minX=-110.0,maxX=-100.0,minY=-90.0,maxY=-90.0)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([CDF4E70BC4C20419:69B5EDAA7D26DA74]:0)
   [junit4]    > 	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
   [junit4]    > 	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
   [junit4]    > 	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:62)
   [junit4]    > 	at org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations(TestBBoxStrategy.java:97)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:853)
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestBBoxStrategy -Dtests.method=testOperations -Dtests.seed=CDF4E70BC4C20419 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_KW -Dtests.timezone=Etc/GMT-8 -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 0.19s J1 | TestBBoxStrategy.testOperations {#7 seed=[CDF4E70BC4C20419:CBD813564C025EDC]} <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: [Contains] Shouldn't match I#0:Rect(minX=-150.0,maxX=-50.0,minY=-70.0,maxY=-70.0) Q:Rect(minX=-150.0,maxX=-130.0,minY=-70.0,maxY=-70.0)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([CDF4E70BC4C20419:CBD813564C025EDC]:0)
   [junit4]    > 	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
   [junit4]    > 	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
   [junit4]    > 	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:62)
   [junit4]    > 	at org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations(TestBBoxStrategy.java:97)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:853)
   [junit4]   2> ??? 09, 2014 2:53:35 ? org.apache.lucene.spatial.StrategyTestCase executeQueries
   [junit4]   2> INFO: testing queried for strategy BBoxStrategy field:bbox ctx=SpatialContext.GEO
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build/spatial/test/J1/./temp/lucene.spatial.bbox.TestBBoxStrategy-CDF4E70BC4C20419-001
   [junit4]   2> NOTE: test params are: codec=Lucene40, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {bbox__minY=DFR I(n)B3(800.0), bbox__maxX=IB SPL-D1, bbox__maxY=IB LL-D1, bbox__xdl=DFR GZ(0.3), bbox__minX=DFR G1}, locale=ar_KW, timezone=Etc/GMT-8
   [junit4]   2> NOTE: Linux 3.8.0-42-generic x86/IBM Corporation 1.7.0 (32-bit)/cpus=8,threads=1,free=24357064,total=56819712
   [junit4]   2> NOTE: All tests run in this JVM: [JtsPolygonTest, TestTermQueryPrefixGridStrategy, TestPointVectorStrategy, DistanceStrategyTest, SpatialPrefixTreeTest, TestBBoxStrategy]
   [junit4] Completed on J1 in 10.26s, 30 tests, 2 failures, 1 skipped <<< FAILURES!

[...truncated 20 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:460: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build.xml:544: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:2019: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/module-build.xml:60: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1298: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:922: There were test failures: 16 suites, 143 tests, 2 failures, 13 ignored (2 assumptions)

Total time: 39 minutes 55 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any