You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/02/17 05:54:15 UTC

[JENKINS] Lucene-Solr-trunk-Linux (32bit/ibm-j9-jdk7) - Build # 9503 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9503/
Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}

7 tests failed.
REGRESSION:  org.apache.solr.analytics.expression.ExpressionTest.multiplyTest

Error Message:
Caught exception in getStatResult, xPath = /response/lst[@name='stats']/lst[@name='mr']/double[@name='sum']  raw data: <?xml version="1.0" encoding="UTF-8"?> <response>  <lst name="responseHeader">   <int name="status">0</int>   <int name="QTime">12</int> </lst> <result name="response" numFound="100" start="0"> </result> <lst name="stats">   <lst name="stats_fields"/> </lst> <lst name="stats">   <lst name="pr">     <long name="count">100</long>     <double name="mc">1.2729436151389237E146</double>     <double name="mean">28.91</double>     <double name="su">3.9603293116911423E124</double>     <double name="sum">2891.0</double>     <long name="unique">36</long>   </lst>   <lst name="dr">     <long name="count">100</long>     <null name="mc"/>     <null name="mean"/>     <null name="su"/>     <null name="sum"/>     <null name="unique"/>   </lst>   <lst name="dmr">     <str name="cma">+2MONTHS</str>     <str name="cme">+2YEARS</str>     <date name="dmma">1812-02-29T23:59:59Z</date>     <date name="dmme">1807-12-31T23:59:59Z</date>     <date name="max">1811-12-31T23:59:59Z</date>     <date name="median">1805-12-31T23:59:59Z</date>   </lst>   <lst name="cr">     <str name="ccmax">this is the secondstr9</str>     <str name="ccmin">this is the firststr0</str>     <str name="csmax">this is the second</str>     <str name="csmin">this is the first</str>     <str name="max">str9</str>     <str name="min">str0</str>   </lst>   <lst name="cnr">     <double name="c10">10.0</double>     <double name="c8">8.0</double>   </lst>   <lst name="rr">     <str name="max">str9</str>     <str name="min">str0</str>     <str name="rmax">9rts</str>     <str name="rmin">0rts</str>   </lst>   <lst name="csr">     <str name="cs1">this is the first</str>     <str name="cs2">this is the second</str>     <str name="cs3">this is the third</str>   </lst>   <lst name="avr">     <double name="c">100.0</double>     <long name="count">100</long>     <double name="s">2891.0</double>     <double name="sum">2891.0</double>   </lst>   <lst name="ar">     <long name="count">100</long>     <null name="mcm"/>     <null name="mean"/>     <null name="median"/>     <null name="su"/>     <null name="sum"/>     <null name="unique"/>   </lst>   <lst name="nr">     <double name="c">-100.0</double>     <long name="count">100</long>     <double name="s">-2891.0</double>     <double name="sum">2891.0</double>   </lst>   <lst name="cdr">     <date name="cd1">1800-12-31T23:59:59Z</date>     <date name="cd2">1804-06-30T23:59:59Z</date>     <str name="cs1">1800-12-31T23:59:59Z</str>     <str name="cs2">1804-06-30T23:59:59Z</str>   </lst>   <lst name="mr">     <long name="count">100</long>     <null name="mcm"/>     <null name="mean"/>     <null name="median"/>     <null name="su"/>     <null name="sum"/>     <null name="unique"/>   </lst> </lst> </response> 

Stack Trace:
java.lang.AssertionError: Caught exception in getStatResult, xPath = /response/lst[@name='stats']/lst[@name='mr']/double[@name='sum'] 
raw data: <?xml version="1.0" encoding="UTF-8"?>
<response>

<lst name="responseHeader">
  <int name="status">0</int>
  <int name="QTime">12</int>
</lst>
<result name="response" numFound="100" start="0">
</result>
<lst name="stats">
  <lst name="stats_fields"/>
</lst>
<lst name="stats">
  <lst name="pr">
    <long name="count">100</long>
    <double name="mc">1.2729436151389237E146</double>
    <double name="mean">28.91</double>
    <double name="su">3.9603293116911423E124</double>
    <double name="sum">2891.0</double>
    <long name="unique">36</long>
  </lst>
  <lst name="dr">
    <long name="count">100</long>
    <null name="mc"/>
    <null name="mean"/>
    <null name="su"/>
    <null name="sum"/>
    <null name="unique"/>
  </lst>
  <lst name="dmr">
    <str name="cma">+2MONTHS</str>
    <str name="cme">+2YEARS</str>
    <date name="dmma">1812-02-29T23:59:59Z</date>
    <date name="dmme">1807-12-31T23:59:59Z</date>
    <date name="max">1811-12-31T23:59:59Z</date>
    <date name="median">1805-12-31T23:59:59Z</date>
  </lst>
  <lst name="cr">
    <str name="ccmax">this is the secondstr9</str>
    <str name="ccmin">this is the firststr0</str>
    <str name="csmax">this is the second</str>
    <str name="csmin">this is the first</str>
    <str name="max">str9</str>
    <str name="min">str0</str>
  </lst>
  <lst name="cnr">
    <double name="c10">10.0</double>
    <double name="c8">8.0</double>
  </lst>
  <lst name="rr">
    <str name="max">str9</str>
    <str name="min">str0</str>
    <str name="rmax">9rts</str>
    <str name="rmin">0rts</str>
  </lst>
  <lst name="csr">
    <str name="cs1">this is the first</str>
    <str name="cs2">this is the second</str>
    <str name="cs3">this is the third</str>
  </lst>
  <lst name="avr">
    <double name="c">100.0</double>
    <long name="count">100</long>
    <double name="s">2891.0</double>
    <double name="sum">2891.0</double>
  </lst>
  <lst name="ar">
    <long name="count">100</long>
    <null name="mcm"/>
    <null name="mean"/>
    <null name="median"/>
    <null name="su"/>
    <null name="sum"/>
    <null name="unique"/>
  </lst>
  <lst name="nr">
    <double name="c">-100.0</double>
    <long name="count">100</long>
    <double name="s">-2891.0</double>
    <double name="sum">2891.0</double>
  </lst>
  <lst name="cdr">
    <date name="cd1">1800-12-31T23:59:59Z</date>
    <date name="cd2">1804-06-30T23:59:59Z</date>
    <str name="cs1">1800-12-31T23:59:59Z</str>
    <str name="cs2">1804-06-30T23:59:59Z</str>
  </lst>
  <lst name="mr">
    <long name="count">100</long>
    <null name="mcm"/>
    <null name="mean"/>
    <null name="median"/>
    <null name="su"/>
    <null name="sum"/>
    <null name="unique"/>
  </lst>
</lst>
</response>

	at __randomizedtesting.SeedInfo.seed([5347EF7F9CE06CA1:38C6F101A9EE5ACF]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.analytics.AbstractAnalyticsStatsTest.getStatResult(AbstractAnalyticsStatsTest.java:114)
	at org.apache.solr.analytics.expression.ExpressionTest.multiplyTest(ExpressionTest.java:92)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:613)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:780)


REGRESSION:  org.apache.solr.analytics.expression.ExpressionTest.addTest

Error Message:
Caught exception in getStatResult, xPath = /response/lst[@name='stats']/lst[@name='ar']/double[@name='sum']  raw data: <?xml version="1.0" encoding="UTF-8"?> <response>  <lst name="responseHeader">   <int name="status">0</int>   <int name="QTime">12</int> </lst> <result name="response" numFound="100" start="0"> </result> <lst name="stats">   <lst name="stats_fields"/> </lst> <lst name="stats">   <lst name="pr">     <long name="count">100</long>     <double name="mc">1.2729436151389237E146</double>     <double name="mean">28.91</double>     <double name="su">3.9603293116911423E124</double>     <double name="sum">2891.0</double>     <long name="unique">36</long>   </lst>   <lst name="dr">     <long name="count">100</long>     <null name="mc"/>     <null name="mean"/>     <null name="su"/>     <null name="sum"/>     <null name="unique"/>   </lst>   <lst name="dmr">     <str name="cma">+2MONTHS</str>     <str name="cme">+2YEARS</str>     <date name="dmma">1812-02-29T23:59:59Z</date>     <date name="dmme">1807-12-31T23:59:59Z</date>     <date name="max">1811-12-31T23:59:59Z</date>     <date name="median">1805-12-31T23:59:59Z</date>   </lst>   <lst name="cr">     <str name="ccmax">this is the secondstr9</str>     <str name="ccmin">this is the firststr0</str>     <str name="csmax">this is the second</str>     <str name="csmin">this is the first</str>     <str name="max">str9</str>     <str name="min">str0</str>   </lst>   <lst name="cnr">     <double name="c10">10.0</double>     <double name="c8">8.0</double>   </lst>   <lst name="rr">     <str name="max">str9</str>     <str name="min">str0</str>     <str name="rmax">9rts</str>     <str name="rmin">0rts</str>   </lst>   <lst name="csr">     <str name="cs1">this is the first</str>     <str name="cs2">this is the second</str>     <str name="cs3">this is the third</str>   </lst>   <lst name="avr">     <double name="c">100.0</double>     <long name="count">100</long>     <double name="s">2891.0</double>     <double name="sum">2891.0</double>   </lst>   <lst name="ar">     <long name="count">100</long>     <null name="mcm"/>     <null name="mean"/>     <null name="median"/>     <null name="su"/>     <null name="sum"/>     <null name="unique"/>   </lst>   <lst name="nr">     <double name="c">-100.0</double>     <long name="count">100</long>     <double name="s">-2891.0</double>     <double name="sum">2891.0</double>   </lst>   <lst name="cdr">     <date name="cd1">1800-12-31T23:59:59Z</date>     <date name="cd2">1804-06-30T23:59:59Z</date>     <str name="cs1">1800-12-31T23:59:59Z</str>     <str name="cs2">1804-06-30T23:59:59Z</str>   </lst>   <lst name="mr">     <long name="count">100</long>     <null name="mcm"/>     <null name="mean"/>     <null name="median"/>     <null name="su"/>     <null name="sum"/>     <null name="unique"/>   </lst> </lst> </response> 

Stack Trace:
java.lang.AssertionError: Caught exception in getStatResult, xPath = /response/lst[@name='stats']/lst[@name='ar']/double[@name='sum'] 
raw data: <?xml version="1.0" encoding="UTF-8"?>
<response>

<lst name="responseHeader">
  <int name="status">0</int>
  <int name="QTime">12</int>
</lst>
<result name="response" numFound="100" start="0">
</result>
<lst name="stats">
  <lst name="stats_fields"/>
</lst>
<lst name="stats">
  <lst name="pr">
    <long name="count">100</long>
    <double name="mc">1.2729436151389237E146</double>
    <double name="mean">28.91</double>
    <double name="su">3.9603293116911423E124</double>
    <double name="sum">2891.0</double>
    <long name="unique">36</long>
  </lst>
  <lst name="dr">
    <long name="count">100</long>
    <null name="mc"/>
    <null name="mean"/>
    <null name="su"/>
    <null name="sum"/>
    <null name="unique"/>
  </lst>
  <lst name="dmr">
    <str name="cma">+2MONTHS</str>
    <str name="cme">+2YEARS</str>
    <date name="dmma">1812-02-29T23:59:59Z</date>
    <date name="dmme">1807-12-31T23:59:59Z</date>
    <date name="max">1811-12-31T23:59:59Z</date>
    <date name="median">1805-12-31T23:59:59Z</date>
  </lst>
  <lst name="cr">
    <str name="ccmax">this is the secondstr9</str>
    <str name="ccmin">this is the firststr0</str>
    <str name="csmax">this is the second</str>
    <str name="csmin">this is the first</str>
    <str name="max">str9</str>
    <str name="min">str0</str>
  </lst>
  <lst name="cnr">
    <double name="c10">10.0</double>
    <double name="c8">8.0</double>
  </lst>
  <lst name="rr">
    <str name="max">str9</str>
    <str name="min">str0</str>
    <str name="rmax">9rts</str>
    <str name="rmin">0rts</str>
  </lst>
  <lst name="csr">
    <str name="cs1">this is the first</str>
    <str name="cs2">this is the second</str>
    <str name="cs3">this is the third</str>
  </lst>
  <lst name="avr">
    <double name="c">100.0</double>
    <long name="count">100</long>
    <double name="s">2891.0</double>
    <double name="sum">2891.0</double>
  </lst>
  <lst name="ar">
    <long name="count">100</long>
    <null name="mcm"/>
    <null name="mean"/>
    <null name="median"/>
    <null name="su"/>
    <null name="sum"/>
    <null name="unique"/>
  </lst>
  <lst name="nr">
    <double name="c">-100.0</double>
    <long name="count">100</long>
    <double name="s">-2891.0</double>
    <double name="sum">2891.0</double>
  </lst>
  <lst name="cdr">
    <date name="cd1">1800-12-31T23:59:59Z</date>
    <date name="cd2">1804-06-30T23:59:59Z</date>
    <str name="cs1">1800-12-31T23:59:59Z</str>
    <str name="cs2">1804-06-30T23:59:59Z</str>
  </lst>
  <lst name="mr">
    <long name="count">100</long>
    <null name="mcm"/>
    <null name="mean"/>
    <null name="median"/>
    <null name="su"/>
    <null name="sum"/>
    <null name="unique"/>
  </lst>
</lst>
</response>

	at __randomizedtesting.SeedInfo.seed([5347EF7F9CE06CA1:A56B42A1DD2F6BE6]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.analytics.AbstractAnalyticsStatsTest.getStatResult(AbstractAnalyticsStatsTest.java:114)
	at org.apache.solr.analytics.expression.ExpressionTest.addTest(ExpressionTest.java:78)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:613)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:780)


REGRESSION:  org.apache.solr.analytics.expression.ExpressionTest.divideTest

Error Message:
Caught exception in getStatResult, xPath = /response/lst[@name='stats']/lst[@name='dr']/double[@name='sum']  raw data: <?xml version="1.0" encoding="UTF-8"?> <response>  <lst name="responseHeader">   <int name="status">0</int>   <int name="QTime">12</int> </lst> <result name="response" numFound="100" start="0"> </result> <lst name="stats">   <lst name="stats_fields"/> </lst> <lst name="stats">   <lst name="pr">     <long name="count">100</long>     <double name="mc">1.2729436151389237E146</double>     <double name="mean">28.91</double>     <double name="su">3.9603293116911423E124</double>     <double name="sum">2891.0</double>     <long name="unique">36</long>   </lst>   <lst name="dr">     <long name="count">100</long>     <null name="mc"/>     <null name="mean"/>     <null name="su"/>     <null name="sum"/>     <null name="unique"/>   </lst>   <lst name="dmr">     <str name="cma">+2MONTHS</str>     <str name="cme">+2YEARS</str>     <date name="dmma">1812-02-29T23:59:59Z</date>     <date name="dmme">1807-12-31T23:59:59Z</date>     <date name="max">1811-12-31T23:59:59Z</date>     <date name="median">1805-12-31T23:59:59Z</date>   </lst>   <lst name="cr">     <str name="ccmax">this is the secondstr9</str>     <str name="ccmin">this is the firststr0</str>     <str name="csmax">this is the second</str>     <str name="csmin">this is the first</str>     <str name="max">str9</str>     <str name="min">str0</str>   </lst>   <lst name="cnr">     <double name="c10">10.0</double>     <double name="c8">8.0</double>   </lst>   <lst name="rr">     <str name="max">str9</str>     <str name="min">str0</str>     <str name="rmax">9rts</str>     <str name="rmin">0rts</str>   </lst>   <lst name="csr">     <str name="cs1">this is the first</str>     <str name="cs2">this is the second</str>     <str name="cs3">this is the third</str>   </lst>   <lst name="avr">     <double name="c">100.0</double>     <long name="count">100</long>     <double name="s">2891.0</double>     <double name="sum">2891.0</double>   </lst>   <lst name="ar">     <long name="count">100</long>     <null name="mcm"/>     <null name="mean"/>     <null name="median"/>     <null name="su"/>     <null name="sum"/>     <null name="unique"/>   </lst>   <lst name="nr">     <double name="c">-100.0</double>     <long name="count">100</long>     <double name="s">-2891.0</double>     <double name="sum">2891.0</double>   </lst>   <lst name="cdr">     <date name="cd1">1800-12-31T23:59:59Z</date>     <date name="cd2">1804-06-30T23:59:59Z</date>     <str name="cs1">1800-12-31T23:59:59Z</str>     <str name="cs2">1804-06-30T23:59:59Z</str>   </lst>   <lst name="mr">     <long name="count">100</long>     <null name="mcm"/>     <null name="mean"/>     <null name="median"/>     <null name="su"/>     <null name="sum"/>     <null name="unique"/>   </lst> </lst> </response> 

Stack Trace:
java.lang.AssertionError: Caught exception in getStatResult, xPath = /response/lst[@name='stats']/lst[@name='dr']/double[@name='sum'] 
raw data: <?xml version="1.0" encoding="UTF-8"?>
<response>

<lst name="responseHeader">
  <int name="status">0</int>
  <int name="QTime">12</int>
</lst>
<result name="response" numFound="100" start="0">
</result>
<lst name="stats">
  <lst name="stats_fields"/>
</lst>
<lst name="stats">
  <lst name="pr">
    <long name="count">100</long>
    <double name="mc">1.2729436151389237E146</double>
    <double name="mean">28.91</double>
    <double name="su">3.9603293116911423E124</double>
    <double name="sum">2891.0</double>
    <long name="unique">36</long>
  </lst>
  <lst name="dr">
    <long name="count">100</long>
    <null name="mc"/>
    <null name="mean"/>
    <null name="su"/>
    <null name="sum"/>
    <null name="unique"/>
  </lst>
  <lst name="dmr">
    <str name="cma">+2MONTHS</str>
    <str name="cme">+2YEARS</str>
    <date name="dmma">1812-02-29T23:59:59Z</date>
    <date name="dmme">1807-12-31T23:59:59Z</date>
    <date name="max">1811-12-31T23:59:59Z</date>
    <date name="median">1805-12-31T23:59:59Z</date>
  </lst>
  <lst name="cr">
    <str name="ccmax">this is the secondstr9</str>
    <str name="ccmin">this is the firststr0</str>
    <str name="csmax">this is the second</str>
    <str name="csmin">this is the first</str>
    <str name="max">str9</str>
    <str name="min">str0</str>
  </lst>
  <lst name="cnr">
    <double name="c10">10.0</double>
    <double name="c8">8.0</double>
  </lst>
  <lst name="rr">
    <str name="max">str9</str>
    <str name="min">str0</str>
    <str name="rmax">9rts</str>
    <str name="rmin">0rts</str>
  </lst>
  <lst name="csr">
    <str name="cs1">this is the first</str>
    <str name="cs2">this is the second</str>
    <str name="cs3">this is the third</str>
  </lst>
  <lst name="avr">
    <double name="c">100.0</double>
    <long name="count">100</long>
    <double name="s">2891.0</double>
    <double name="sum">2891.0</double>
  </lst>
  <lst name="ar">
    <long name="count">100</long>
    <null name="mcm"/>
    <null name="mean"/>
    <null name="median"/>
    <null name="su"/>
    <null name="sum"/>
    <null name="unique"/>
  </lst>
  <lst name="nr">
    <double name="c">-100.0</double>
    <long name="count">100</long>
    <double name="s">-2891.0</double>
    <double name="sum">2891.0</double>
  </lst>
  <lst name="cdr">
    <date name="cd1">1800-12-31T23:59:59Z</date>
    <date name="cd2">1804-06-30T23:59:59Z</date>
    <str name="cs1">1800-12-31T23:59:59Z</str>
    <str name="cs2">1804-06-30T23:59:59Z</str>
  </lst>
  <lst name="mr">
    <long name="count">100</long>
    <null name="mcm"/>
    <null name="mean"/>
    <null name="median"/>
    <null name="su"/>
    <null name="sum"/>
    <null name="unique"/>
  </lst>
</lst>
</response>

	at __randomizedtesting.SeedInfo.seed([5347EF7F9CE06CA1:B4A0489B8220E77]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.analytics.AbstractAnalyticsStatsTest.getStatResult(AbstractAnalyticsStatsTest.java:114)
	at org.apache.solr.analytics.expression.ExpressionTest.divideTest(ExpressionTest.java:106)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:613)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:780)


REGRESSION:  org.apache.solr.handler.component.DistributedQueryElevationComponentTest.testDistribSearch

Error Message:
.responseHeader.params.sort!=version (unordered or missing)

Stack Trace:
junit.framework.AssertionFailedError: .responseHeader.params.sort!=version (unordered or missing)
	at __randomizedtesting.SeedInfo.seed([5347EF7F9CE06CA1:D2A16167EBBF0C9D]:0)
	at junit.framework.Assert.fail(Assert.java:50)
	at org.apache.solr.BaseDistributedSearchTestCase.compareSolrResponses(BaseDistributedSearchTestCase.java:843)
	at org.apache.solr.BaseDistributedSearchTestCase.compareResponses(BaseDistributedSearchTestCase.java:862)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:565)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:545)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:524)
	at org.apache.solr.handler.component.DistributedQueryElevationComponentTest.doTest(DistributedQueryElevationComponentTest.java:81)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:613)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:780)


REGRESSION:  org.apache.solr.handler.component.TermVectorComponentDistributedTest.testDistribSearch

Error Message:
.termVectors.0.test_offtv!=test_basictv (unordered or missing)

Stack Trace:
junit.framework.AssertionFailedError: .termVectors.0.test_offtv!=test_basictv (unordered or missing)
	at __randomizedtesting.SeedInfo.seed([5347EF7F9CE06CA1:D2A16167EBBF0C9D]:0)
	at junit.framework.Assert.fail(Assert.java:50)
	at org.apache.solr.BaseDistributedSearchTestCase.compareSolrResponses(BaseDistributedSearchTestCase.java:843)
	at org.apache.solr.BaseDistributedSearchTestCase.compareResponses(BaseDistributedSearchTestCase.java:862)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:565)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:545)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:524)
	at org.apache.solr.handler.component.TermVectorComponentDistributedTest.doTest(TermVectorComponentDistributedTest.java:137)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:876)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:613)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:780)


REGRESSION:  org.apache.solr.update.AutoCommitTest.testMaxTime

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
	at __randomizedtesting.SeedInfo.seed([5347EF7F9CE06CA1:C9B3929D027AF09D]:0)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:711)
	at org.apache.solr.update.AutoCommitTest.testMaxTime(AutoCommitTest.java:227)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:613)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:780)
Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=0]
	xml response was: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">2312</int></lst><result name="response" numFound="1" start="0"><doc><int name="id">529</int><int name="range_facet_si">529</int><arr name="range_facet_l"><long>529</long></arr><arr name="range_facet_sl"><long>529</long></arr><arr name="field_t"><str>what's inside?</str></arr><arr name="subject"><str>info</str></arr><long name="_version_">1460258869595865088</long><int name="intDefault">42</int><date name="timestamp">2014-02-17T04:29:39.238Z</date><arr name="multiDefault"><str>muLti-Default</str></arr></doc></result>
</response>

	request was:rows=20&start=0&version=2.2&qt=standard&q=id:529
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:704)
	... 41 more


REGRESSION:  org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.testDupeDetection

Error Message:
expected:<1> but was:<2>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<2>
	at __randomizedtesting.SeedInfo.seed([5347EF7F9CE06CA1:87714B79FA7CFDD]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.checkNumDocs(SignatureUpdateProcessorFactoryTest.java:71)
	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.testDupeDetection(SignatureUpdateProcessorFactoryTest.java:119)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:613)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:780)




Build Log:
[...truncated 10156 lines...]
   [junit4] Suite: org.apache.solr.handler.component.TermVectorComponentDistributedTest
   [junit4]   2> 306960 T903 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
   [junit4]   2> 306961 T903 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /hm/g
   [junit4]   2> 306964 T903 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 306965 T903 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 306968 T903 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:43150
   [junit4]   2> 306969 T903 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 306969 T903 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 306970 T903 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr
   [junit4]   2> 306971 T903 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/'
   [junit4]   2> 306979 T903 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/solr.xml
   [junit4]   2> 307012 T903 oasc.CoreContainer.<init> New CoreContainer -73117230
   [junit4]   2> 307012 T903 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/]
   [junit4]   2> 307014 T903 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 307014 T903 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 307015 T903 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 307015 T903 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 307016 T903 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 307017 T903 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 307017 T903 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 307018 T903 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 307018 T903 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 307022 T903 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 307023 T903 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 307024 T903 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 307027 T913 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1
   [junit4]   2> 307027 T913 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/'
   [junit4]   2> 307028 T913 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 307029 T913 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 307069 T913 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 307136 T913 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 307137 T913 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 307144 T913 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 307474 T913 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 307480 T913 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 307483 T913 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 307490 T913 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 307494 T913 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 307498 T913 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 307498 T913 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 307499 T913 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 307499 T913 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 307500 T913 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 307500 T913 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 307501 T913 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 307501 T913 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392611124577/control/data/
   [junit4]   2> 307501 T913 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@addeb9a5
   [junit4]   2> 307502 T913 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392611124577/control/data
   [junit4]   2> 307503 T913 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392611124577/control/data/index/
   [junit4]   2> 307503 T913 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392611124577/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 307503 T913 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392611124577/control/data/index
   [junit4]   2> 307504 T913 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=31, maxMergeAtOnceExplicit=41, maxMergedSegmentMB=54.1201171875, floorSegmentMB=1.9013671875, forceMergeDeletesPctAllowed=23.683411958514565, segmentsPerTier=28.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 307505 T913 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@6d2d91b8 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@d8e27555)),segFN=segments_1,generation=1}
   [junit4]   2> 307505 T913 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 307505 T913 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
   [junit4]   2> 307508 T913 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 307508 T913 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 307509 T913 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 307509 T913 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 307509 T913 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 307510 T913 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 307510 T913 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 307511 T913 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 307511 T913 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 307511 T913 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 307512 T913 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 307512 T913 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 307513 T913 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 307513 T913 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 307513 T913 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 307514 T913 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 307514 T913 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
   [junit4]   2> 307515 T913 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 307515 T913 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 307516 T913 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 307516 T913 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 307517 T913 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 307517 T913 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 307517 T913 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 307518 T913 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 307518 T913 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 307519 T913 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 307519 T913 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 307520 T913 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 307520 T913 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 307521 T913 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 307521 T913 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 307521 T913 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 307522 T913 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 307522 T913 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 307524 T913 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 307525 T913 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 307528 T913 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 307528 T913 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 307529 T913 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=45, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.44211191143953454]
   [junit4]   2> 307529 T913 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@6d2d91b8 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@d8e27555)),segFN=segments_1,generation=1}
   [junit4]   2> 307530 T913 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 307530 T913 oass.SolrIndexSearcher.<init> Opening Searcher@2755d97e[collection1] main
   [junit4]   2> 307531 T913 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 307535 T913 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 307557 T914 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2755d97e[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 307561 T913 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 307561 T903 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 307562 T903 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 307563 T903 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 307566 T903 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:44911
   [junit4]   2> 307566 T903 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 307567 T903 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 307568 T903 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr
   [junit4]   2> 307568 T903 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/'
   [junit4]   2> 307577 T903 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/solr.xml
   [junit4]   2> 307610 T903 oasc.CoreContainer.<init> New CoreContainer -1585713983
   [junit4]   2> 307611 T903 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/]
   [junit4]   2> 307612 T903 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 307612 T903 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 307613 T903 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 307614 T903 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 307614 T903 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 307615 T903 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 307615 T903 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 307616 T903 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 307616 T903 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 307620 T903 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 307621 T903 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 307621 T903 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 307625 T925 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1
   [junit4]   2> 307625 T925 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/'
   [junit4]   2> 307626 T925 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 307626 T925 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 307664 T925 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 307726 T925 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 307727 T925 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 307733 T925 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 308056 T925 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 308064 T925 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 308069 T925 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 308077 T925 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 308080 T925 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 308084 T925 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 308085 T925 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 308085 T925 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 308085 T925 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 308086 T925 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 308086 T925 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 308087 T925 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 308087 T925 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392611124577/shard0/data/
   [junit4]   2> 308087 T925 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@addeb9a5
   [junit4]   2> 308088 T925 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392611124577/shard0/data
   [junit4]   2> 308089 T925 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392611124577/shard0/data/index/
   [junit4]   2> 308089 T925 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392611124577/shard0/data/index' doesn't exist. Creating new index...
   [junit4]   2> 308090 T925 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392611124577/shard0/data/index
   [junit4]   2> 308090 T925 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=31, maxMergeAtOnceExplicit=41, maxMergedSegmentMB=54.1201171875, floorSegmentMB=1.9013671875, forceMergeDeletesPctAllowed=23.683411958514565, segmentsPerTier=28.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 308091 T925 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@bfdb3ce9 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@a2a62d5a)),segFN=segments_1,generation=1}
   [junit4]   2> 308091 T925 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 308092 T925 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
   [junit4]   2> 308095 T925 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 308095 T925 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 308095 T925 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 308096 T925 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 308096 T925 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 308097 T925 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 308097 T925 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 308097 T925 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 308098 T925 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 308098 T925 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 308099 T925 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 308099 T925 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 308099 T925 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 308100 T925 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 308100 T925 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 308101 T925 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 308102 T925 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
   [junit4]   2> 308102 T925 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 308102 T925 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 308103 T925 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 308104 T925 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 308104 T925 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 308104 T925 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 308105 T925 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 308105 T925 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 308106 T925 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 308106 T925 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 308107 T925 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 308108 T925 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 308108 T925 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 308109 T925 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 308109 T925 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 308110 T925 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 308110 T925 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 308111 T925 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 308112 T925 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 308114 T925 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 308116 T925 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 308116 T925 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 308117 T925 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=45, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.44211191143953454]
   [junit4]   2> 308118 T925 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@bfdb3ce9 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@a2a62d5a)),segFN=segments_1,generation=1}
   [junit4]   2> 308118 T925 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 308119 T925 oass.SolrIndexSearcher.<init> Opening Searcher@4c0fd806[collection1] main
   [junit4]   2> 308119 T925 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 308194 T925 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 308216 T926 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4c0fd806[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 308222 T925 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 308222 T903 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 308223 T903 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> ASYNC  NEW_CORE C2107 name=collection1 org.apache.solr.core.SolrCore@9799b4a0
   [junit4]   2> 308229 T909 C2107 oasup.LogUpdateProcessor.finish [collection1] webapp=/hm/g path=/update params={version=2&wt=javabin} {add=[0 (1460258603885658112)]} 0 4
   [junit4]   2> ASYNC  NEW_CORE C2108 name=collection1 org.apache.solr.core.SolrCore@f5015255
   [junit4]   2> 308233 T923 C2108 oasup.LogUpdateProcessor.finish [collection1] webapp=/hm/g path=/update params={version=2&wt=javabin} {add=[0 (1460258603891949568)]} 0 2
   [junit4]   2> 308238 T910 C2107 oasup.LogUpdateProcessor.finish [collection1] webapp=/hm/g path=/update params={version=2&wt=javabin} {add=[1 (1460258603897192448)]} 0 2
   [junit4]   2> 308242 T924 C2108 oasup.LogUpdateProcessor.finish [collection1] webapp=/hm/g path=/update params={version=2&wt=javabin} {add=[1 (1460258603901386752)]} 0 2
   [junit4]   2> 308246 T911 C2107 oasup.LogUpdateProcessor.finish [collection1] webapp=/hm/g path=/update params={version=2&wt=javabin} {add=[2 (1460258603906629632)]} 0 2
   [junit4]   2> 308249 T921 C2108 oasup.LogUpdateProcessor.finish [collection1] webapp=/hm/g path=/update params={version=2&wt=javabin} {add=[2 (1460258603909775360)]} 0 1
   [junit4]   2> 308253 T912 C2107 oasup.LogUpdateProcessor.finish [collection1] webapp=/hm/g path=/update params={version=2&wt=javabin} {add=[3 (1460258603913969664)]} 0 1
   [junit4]   2> 308256 T922 C2108 oasup.LogUpdateProcessor.finish [collection1] webapp=/hm/g path=/update params={version=2&wt=javabin} {add=[3 (1460258603917115392)]} 0 1
   [junit4]   2> 308259 T909 C2107 oasup.LogUpdateProcessor.finish [collection1] webapp=/hm/g path=/update params={version=2&wt=javabin} {add=[4 (1460258603920261120)]} 0 1
   [junit4]   2> 308261 T923 C2108 oasup.LogUpdateProcessor.finish [collection1] webapp=/hm/g path=/update params={version=2&wt=javabin} {add=[4 (1460258603923406848)]} 0 0
   [junit4]   2> 308264 T910 C2107 oasup.LogUpdateProcessor.finish [collection1] webapp=/hm/g path=/update params={version=2&wt=javabin} {add=[5 (1460258603925504000)]} 0 1
   [junit4]   2> 308267 T924 C2108 oasup.LogUpdateProcessor.finish [collection1] webapp=/hm/g path=/update params={version=2&wt=javabin} {add=[5 (1460258603928649728)]} 0 1
   [junit4]   2> 308269 T911 C2107 oasup.LogUpdateProcessor.finish [collection1] webapp=/hm/g path=/update params={version=2&wt=javabin} {add=[6 (1460258603931795456)]} 0 0
   [junit4]   2> 308272 T921 C2108 oasup.LogUpdateProcessor.finish [collection1] webapp=/hm/g path=/update params={version=2&wt=javabin} {add=[6 (1460258603933892608)]} 0 1
   [junit4]   2> 308275 T912 C2107 oasup.LogUpdateProcessor.finish [collection1] webapp=/hm/g path=/update params={version=2&wt=javabin} {add=[7 (1460258603937038336)]} 0 1
   [junit4]   2> 308277 T922 C2108 oasup.LogUpdateProcessor.finish [collection1] webapp=/hm/g path=/update params={version=2&wt=javabin} {add=[7 (1460258603940184064)]} 0 0
   [junit4]   2> 308280 T909 C2107 oasup.LogUpdateProcessor.finish [collection1] webapp=/hm/g path=/update params={version=2&wt=javabin} {add=[8 (1460258603943329792)]} 0 1
   [junit4]   2> 308283 T923 C2108 oasup.LogUpdateProcessor.finish [collection1] webapp=/hm/g path=/update params={version=2&wt=javabin} {add=[8 (1460258603945426944)]} 0 1
   [junit4]   2> 308285 T910 C2107 oasup.LogUpdateProcessor.finish [collection1] webapp=/hm/g path=/update params={version=2&wt=javabin} {add=[9 (1460258603948572672)]} 0 0
   [junit4]   2> 308288 T924 C2108 oasup.LogUpdateProcessor.finish [collection1] webapp=/hm/g path=/update params={version=2&wt=javabin} {add=[9 (1460258603951718400)]} 0 0
   [junit4]   2> 308290 T911 C2107 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 308297 T911 C2107 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@6d2d91b8 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@d8e27555)),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@6d2d91b8 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@d8e27555)),segFN=segments_2,generation=2}
   [junit4]   2> 308297 T911 C2107 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 308302 T911 C2107 oass.SolrIndexSearcher.<init> Opening Searcher@b3189099[collection1] main
   [junit4]   2> 308303 T911 C2107 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 308305 T914 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@b3189099[collection1] main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c10)}
   [junit4]   2> 308305 T911 C2107 oasup.LogUpdateProcessor.finish [collection1] webapp=/hm/g path=/update params={version=2&wt=javabin&waitSearcher=true&commit=true&softCommit=false} {commit=} 0 15
   [junit4]   2> 308307 T921 C2108 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 308313 T921 C2108 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@bfdb3ce9 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@a2a62d5a)),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@bfdb3ce9 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@a2a62d5a)),segFN=segments_2,generation=2}
   [junit4]   2> 308314 T921 C2108 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 308317 T921 C2108 oass.SolrIndexSearcher.<init> Opening Searcher@37640eb[collection1] main
   [junit4]   2> 308318 T921 C2108 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 308320 T926 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@37640eb[collection1] main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c10)}
   [junit4]   2> 308321 T921 C2108 oasup.LogUpdateProcessor.finish [collection1] webapp=/hm/g path=/update params={commit=true&waitSearcher=true&version=2&wt=javabin&softCommit=false} {commit=} 0 14
   [junit4]   2> 308328 T912 C2107 oasc.SolrCore.execute [collection1] webapp=/hm/g path=/select params={tv.tf=true&sort=id+desc&version=2&wt=javabin&q=id:0&shards.qt=tvrh&tv=true&qt=tvrh&distrib=false} hits=1 status=0 QTime=5 
   [junit4]   2> 308338 T923 C2108 oasc.SolrCore.execute [collection1] webapp=/hm/g path=/select params={wt=javabin&start=0&fl=id&tv=true&isShard=true&rows=10&fsv=true&qt=tvrh&version=2&sort=id+desc&tv.tf=true&shards.qt=tvrh&q=id:0&shard.url=[ff01::213]:33332/hm/g|http://127.0.0.1:44911/hm/g|[ff01::114]:33332/hm/g&NOW=1392611125943&distrib=false} hits=1 status=0 QTime=0 
   [junit4]   2> 308343 T924 C2108 oasc.SolrCore.execute [collection1] webapp=/hm/g path=/select params={ids=0&distrib=false&qt=tvrh&shard.url=[ff01::213]:33332/hm/g|http://127.0.0.1:44911/hm/g|[ff01::114]:33332/hm/g&isShard=true&tv=true&NOW=1392611125943&shards.qt=tvrh&q=id:0&version=2&wt=javabin&tv.tf=true} status=0 QTime=1 
   [junit4]   2> 308344 T922 C2108 oasc.SolrCore.execute [collection1] webapp=/hm/g path=/select params={tv=true&shards=[ff01::213]:33332/hm/g|http://127.0.0.1:44911/hm/g|[ff01::114]:33332/hm/g&qt=tvrh&sort=id+desc&tv.tf=true&version=2&wt=javabin&q=id:0&shards.qt=tvrh} hits=1 status=0 QTime=13 
   [junit4]   2> 308348 T909 C2107 oasc.SolrCore.execute [collection1] webapp=/hm/g path=/select params={version=2&wt=javabin&tv.tf=true&sort=id+asc&fl=*,score&shards.qt=tvrh&q=id:0&tv=true&distrib=false&tv.fl=test_basictv,test_offtv&qt=tvrh} hits=1 status=0 QTime=1 
   [junit4]   2> 308355 T923 C2108 oasc.SolrCore.execute [collection1] webapp=/hm/g path=/select params={tv.fl=test_basictv,test_offtv&distrib=false&shard.url=[ff01::114]:33332/hm/g|[ff01::213]:33332/hm/g|[ff01::083]:33332/hm/g|http://127.0.0.1:44911/hm/g&NOW=1392611125962&shards.qt=tvrh&q=id:0&version=2&sort=id+asc&tv.tf=true&rows=10&fsv=true&qt=tvrh&tv=true&isShard=true&fl=id,score&wt=javabin&start=0} hits=1 status=0 QTime=1 
   [junit4]   2> 308358 T924 C2108 oasc.SolrCore.execute [collection1] webapp=/hm/g path=/select params={q=id:0&shards.qt=tvrh&tv.tf=true&version=2&tv.fl=test_basictv,test_offtv&distrib=false&ids=0&NOW=1392611125962&shard.url=[ff01::114]:33332/hm/g|[ff01::213]:33332/hm/g|[ff01::083]:33332/hm/g|http://127.0.0.1:44911/hm/g&fl=*,score&wt=javabin&qt=tvrh&tv=true&isShard=true} status=0 QTime=0 
   [junit4]   2> 308360 T921 C2108 oasc.SolrCore.execute [collection1] webapp=/hm/g path=/select params={fl=*,score&shards.qt=tvrh&q=id:0&version=2&wt=javabin&tv.tf=true&sort=id+asc&tv.fl=test_basictv,test_offtv&qt=tvrh&shards=[ff01::114]:33332/hm/g|[ff01::213]:33332/hm/g|[ff01::083]:33332/hm/g|http://127.0.0.1:44911/hm/g&tv=true} hits=1 status=0 QTime=10 
   [junit4]   2> 308364 T910 C2107 oasc.SolrCore.execute [collection1] webapp=/hm/g path=/select params={version=2&wt=javabin&sort=id+asc&tv.tf=true&shards.qt=tvrh&fl=*,score&q=id:0&tv=true&distrib=false&tv.fl=test_basictv&tv.fl=test_offtv&qt=tvrh} hits=1 status=0 QTime=1 
   [junit4]   2> 308368 T923 C2108 oasc.SolrCore.execute [collection1] webapp=/hm/g path=/select params={fl=id,score&start=0&wt=javabin&fsv=true&qt=tvrh&rows=10&tv=true&isShard=true&q=id:0&shards.qt=tvrh&sort=id+asc&tv.tf=true&version=2&tv.fl=test_basictv&tv.fl=test_offtv&distrib=false&NOW=1392611125978&shard.url=[ff01::213]:33332/hm/g|http://127.0.0.1:44911/hm/g|[ff01::083]:33332/hm/g} hits=1 status=0 QTime=0 
   [junit4]   2> 308372 T921 C2108 oasc.SolrCore.execute [collection1] webapp=/hm/g path=/select params={wt=javabin&fl=*,score&tv=true&isShard=true&qt=tvrh&version=2&tv.tf=true&shards.qt=tvrh&q=id:0&shard.url=[ff01::213]:33332/hm/g|http://127.0.0.1:44911/hm/g|[ff01::083]:33332/hm/g&NOW=1392611125978&ids=0&tv.fl=test_basictv&tv.fl=test_offtv&distrib=false} status=0 QTime=1 
   [junit4]   2> 308373 T922 C2108 oasc.SolrCore.execute [collection1] webapp=/hm/g path=/select params={fl=*,score&shards.qt=tvrh&q=id:0&version=2&wt=javabin&tv.tf=true&sort=id+asc&tv.fl=test_basictv&tv.fl=test_offtv&qt=tvrh&shards=[ff01::213]:33332/hm/g|http://127.0.0.1:44911/hm/g|[ff01::083]:33332/hm/g&tv=true} hits=1 status=0 QTime=7 
   [junit4]   2> 308374 T903 oas.BaseDistributedSearchTestCase.compareSolrResponses ERROR Mismatched responses:
   [junit4]   2> 	{responseHeader={status=0,QTime=7},response={numFound=1,start=0,maxScore=2.609438,docs=[SolrDocument{id=0, range_facet_si=0, range_facet_l=[0], range_facet_sl=[0], test_posofftv=[This is a title and another title], test_basictv=[This is a title and another title], test_notv=[This is a title and another title], test_postv=[This is a title and another title], test_offtv=[This is a title and another title], _version_=1460258603891949568, in

[...truncated too long message...]

9 T5360 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 38
   [junit4]   2> 1537960 T5360 oass.SolrIndexSearcher.<init> Opening Searcher@84594ccc[collection1] main
   [junit4]   2> 1537960 T5360 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 1537961 T5361 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1537961 T5361 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 1537961 T5350 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@84594ccc[collection1] main{StandardDirectoryReader(segments_11:249:nrt _2h(5.0):c1 _2j(5.0):c1)}
   [junit4]   2> 1537963 T5361 oass.SolrIndexSearcher.<init> Opening Searcher@86b8892f[collection1] main
   [junit4]   2> 1537964 T5361 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 1537964 T5359 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1537965 T5359 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@21463b2f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@e2266e9a)),segFN=segments_12,generation=38}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@21463b2f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@e2266e9a)),segFN=segments_13,generation=39}
   [junit4]   2> 1537965 T5359 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 39
   [junit4]   2> 1537965 T5350 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@86b8892f[collection1] main{StandardDirectoryReader(segments_12:253:nrt _2k(5.0):c1)}
   [junit4]   2> 1537966 T5359 oass.SolrIndexSearcher.<init> Opening Searcher@dc8b2e63[collection1] main
   [junit4]   2> 1537967 T5359 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 1537968 T5350 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@dc8b2e63[collection1] main{StandardDirectoryReader(segments_12:253:nrt _2k(5.0):c1)}
   [junit4]   2> ASYNC  NEW_CORE C3858 name=collection1 org.apache.solr.core.SolrCore@b58b0db4
   [junit4]   2> 1537969 T5348 C3858 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1537970 T5348 C3858 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 1537970 T5348 C3858 oass.SolrIndexSearcher.<init> Opening Searcher@11c8533a[collection1] main
   [junit4]   2> 1537971 T5348 C3858 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 1537972 T5350 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@11c8533a[collection1] main{StandardDirectoryReader(segments_12:253:nrt _2k(5.0):c1)}
   [junit4]   2> 1537973 T5348 C3858 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 4
   [junit4]   2> 1537976 T5348 oas.SolrTestCaseJ4.tearDown ###Ending testMultiThreaded
   [junit4]   2> 1537980 T5348 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 1537981 T5348 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1410734536
   [junit4]   2> 1537981 T5348 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@b58b0db4
   [junit4]   2> 1537991 T5348 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=42,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=253,cumulative_deletesById=0,cumulative_deletesByQuery=6,cumulative_errors=0}
   [junit4]   2> 1537992 T5348 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 1537992 T5348 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1537992 T5348 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 1537993 T5348 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1537994 T5348 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1537994 T5348 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1392612354013 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1392612354013;done=false>>]
   [junit4]   2> 1537994 T5348 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1392612354013
   [junit4]   2> 1537995 T5348 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1392612354013/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1392612354013/index;done=false>>]
   [junit4]   2> 1537995 T5348 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1392612354013/index
   [junit4]   2> NOTE: test params are: codec=Lucene42, sim=DefaultSimilarity, locale=es, timezone=Europe/Zagreb
   [junit4]   2> NOTE: Linux 3.8.0-35-generic x86/IBM Corporation 1.7.0 (32-bit)/cpus=8,threads=1,free=182746296,total=286130176
   [junit4]   2> NOTE: All tests run in this JVM: [ShardRoutingCustomTest, ConvertedLegacyTest, CoreContainerCoreInitFailuresTest, TestFieldTypeResource, HdfsBasicDistributedZkTest, TestCodecSupport, RAMDirectoryFactoryTest, HighlighterConfigTest, TestJmxIntegration, DebugComponentTest, DeleteReplicaTest, TestCustomSort, TestFastLRUCache, FunctionTest, TestNRTOpen, TestSolr4Spatial, PingRequestHandlerTest, CursorMarkTest, XsltUpdateRequestHandlerTest, HdfsLockFactoryTest, ModifyConfFileTest, ChaosMonkeyNothingIsSafeTest, CachingDirectoryFactoryTest, DirectUpdateHandlerTest, TestMultiCoreConfBootstrap, TestCursorMarkWithoutUniqueKey, IndexSchemaRuntimeFieldTest, TestCollationFieldDocValues, TestInitQParser, MultiTermTest, TermVectorComponentDistributedTest, TestAddFieldRealTimeGet, TestStressLucene, BasicDistributedZk2Test, SolrCmdDistributorTest, DisMaxRequestHandlerTest, FieldAnalysisRequestHandlerTest, ConnectionManagerTest, CurrencyFieldXmlFileTest, BadIndexSchemaTest, TestOmitPositions, TestRandomFaceting, TestFuzzyAnalyzedSuggestions, HdfsSyncSliceTest, DistanceFunctionTest, LeaderElectionIntegrationTest, TestCollationField, TestRangeQuery, NoCacheHeaderTest, DateFieldTest, TestDynamicFieldResource, SpellingQueryConverterTest, TestXIncludeConfig, InfoHandlerTest, SuggesterWFSTTest, TestSchemaVersionResource, TestDistributedGrouping, TestNoOpRegenerator, NotRequiredUniqueKeyTest, TestCSVLoader, TestAtomicUpdateErrorCases, TestSolrCoreProperties, SolrIndexConfigTest, BJQParserTest, QueryFacetTest, TestManagedSchema, BlockDirectoryTest, HardAutoCommitTest, TestIndexingPerformance, BasicZkTest, TestZkChroot, DocValuesMultiTest, TestIBSimilarityFactory, TestFileDictionaryLookup, TestManagedSchemaFieldResource, SimplePostToolTest, OpenExchangeRatesOrgProviderTest, DirectSolrSpellCheckerTest, TestFastWriter, RequestHandlersTest, ChaosMonkeySafeLeaderTest, RemoteQueryErrorTest, SuggesterTSTTest, TestUtils, TestClassNameShortening, TestCoreDiscovery, TestCloudManagedSchema, TestDocBasedVersionConstraints, TestIndexSearcher, PreAnalyzedFieldTest, TestCloudManagedSchemaAddField, CollectionsAPIDistributedZkTest, OverseerTest, FullSolrCloudDistribCmdsTest, ZkCLITest, DistributedSpellCheckComponentTest, TestStressReorder, TestCoreContainer, TestGroupingSearch, TestLazyCores, HighlighterTest, SoftAutoCommitTest, SignatureUpdateProcessorFactoryTest]
   [junit4] Completed on J1 in 1.62s, 6 tests, 1 failure <<< FAILURES!

[...truncated 487 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:453: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:446: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1268: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:901: There were test failures: 372 suites, 1604 tests, 1 error, 6 failures, 35 ignored (7 assumptions)

Total time: 63 minutes 15 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_51) - Build # 9505 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9505/
Java: 32bit/jdk1.7.0_51 -client -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.testDistribSearch

Error Message:
document count mismatch.  control=1952 sum(shards)=1951 cloudClient=1951

Stack Trace:
java.lang.AssertionError: document count mismatch.  control=1952 sum(shards)=1951 cloudClient=1951
	at __randomizedtesting.SeedInfo.seed([AB9C3D0931F47C85:2A7AB31146AB1CB9]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1230)
	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:213)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 10170 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
   [junit4]   2> 265889 T1233 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 265889 T1233 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 265893 T1233 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeyNothingIsSafeTest-1392636921895
   [junit4]   2> 265893 T1233 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 265894 T1234 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 265994 T1233 oasc.ZkTestServer.run start zk server on port:36325
   [junit4]   2> 265995 T1233 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 266063 T1240 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@417c4c name:ZooKeeperConnection Watcher:127.0.0.1:36325 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 266063 T1233 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 266064 T1233 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 266068 T1233 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 266069 T1242 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1712c57 name:ZooKeeperConnection Watcher:127.0.0.1:36325/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 266069 T1233 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 266070 T1233 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 266073 T1233 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 266076 T1233 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 266078 T1233 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 266079 T1233 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 266080 T1233 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 266082 T1233 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 266082 T1233 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 266085 T1233 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 266085 T1233 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 266087 T1233 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 266087 T1233 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 266089 T1233 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 266089 T1233 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 266091 T1233 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 266092 T1233 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 266093 T1233 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 266094 T1233 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 266096 T1233 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 266096 T1233 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 266098 T1233 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 266098 T1233 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 266100 T1233 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 266101 T1233 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 266103 T1233 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 266103 T1233 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 266168 T1233 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 266171 T1233 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:43786
   [junit4]   2> 266172 T1233 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 266172 T1233 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 266172 T1233 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392636922108
   [junit4]   2> 266173 T1233 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392636922108/'
   [junit4]   2> 266184 T1233 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392636922108/solr.xml
   [junit4]   2> 266233 T1233 oasc.CoreContainer.<init> New CoreContainer 18682528
   [junit4]   2> 266233 T1233 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392636922108/]
   [junit4]   2> 266235 T1233 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 266235 T1233 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 266235 T1233 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 266236 T1233 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 266236 T1233 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 266236 T1233 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 266237 T1233 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 266237 T1233 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 266237 T1233 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 266239 T1233 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 266240 T1233 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 266240 T1233 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 266241 T1233 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:36325/solr
   [junit4]   2> 266241 T1233 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 266242 T1233 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 266244 T1253 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b1eef3 name:ZooKeeperConnection Watcher:127.0.0.1:36325 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 266244 T1233 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 266246 T1233 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 266247 T1255 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@692771 name:ZooKeeperConnection Watcher:127.0.0.1:36325/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 266248 T1233 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 266249 T1233 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 266251 T1233 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 266254 T1233 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 266256 T1233 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43786_
   [junit4]   2> 266256 T1233 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43786_
   [junit4]   2> 266258 T1233 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 266260 T1233 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 266262 T1233 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:43786_
   [junit4]   2> 266263 T1233 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 266264 T1233 oasc.Overseer.start Overseer (id=91267853313441795-127.0.0.1:43786_-n_0000000000) starting
   [junit4]   2> 266267 T1233 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 266270 T1257 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 266270 T1233 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 266270 T1257 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 266274 T1233 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 266276 T1233 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 266279 T1256 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 266282 T1258 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 266283 T1258 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 266285 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 266285 T1258 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 266286 T1256 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 266287 T1256 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:43786",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:43786_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 266287 T1256 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 266287 T1256 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 266289 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 266290 T1255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 267285 T1258 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 267286 T1258 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392636922108/collection1
   [junit4]   2> 267286 T1258 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 267287 T1258 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 267287 T1258 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 267288 T1258 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 267288 T1258 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392636922108/collection1/'
   [junit4]   2> 267289 T1258 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392636922108/collection1/lib/classes/' to classloader
   [junit4]   2> 267289 T1258 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392636922108/collection1/lib/README' to classloader
   [junit4]   2> 267336 T1258 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 267380 T1258 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 267382 T1258 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 267389 T1258 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 267738 T1258 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 267740 T1258 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 267742 T1258 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 267746 T1258 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 267780 T1258 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 267791 T1258 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392636922108/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392636921895/control/data/
   [junit4]   2> 267791 T1258 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@16edd1d
   [junit4]   2> 267792 T1258 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392636921895/control/data
   [junit4]   2> 267792 T1258 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392636921895/control/data/index/
   [junit4]   2> 267792 T1258 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392636921895/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 267793 T1258 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392636921895/control/data/index
   [junit4]   2> 267793 T1258 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=26, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 267795 T1258 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392636921895/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 267795 T1258 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 267798 T1258 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 267798 T1258 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 267799 T1258 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 267799 T1258 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 267799 T1258 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 267799 T1258 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 267800 T1258 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 267800 T1258 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 267800 T1258 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 267801 T1258 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 267801 T1258 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 267801 T1258 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 267801 T1258 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 267802 T1258 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 267802 T1258 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 267803 T1258 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 267803 T1258 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 267806 T1258 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 267808 T1258 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 267809 T1258 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 267810 T1258 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=16, maxMergeAtOnceExplicit=14, maxMergedSegmentMB=8.0078125, floorSegmentMB=1.0859375, forceMergeDeletesPctAllowed=21.099702353805615, segmentsPerTier=36.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.79896787184218
   [junit4]   2> 267811 T1258 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392636921895/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 267811 T1258 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 267812 T1258 oass.SolrIndexSearcher.<init> Opening Searcher@842b3d[collection1] main
   [junit4]   2> 267814 T1259 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@842b3d[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 267816 T1258 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 267817 T1233 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 267818 T1233 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 267817 T1262 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:43786 collection:control_collection shard:shard1
   [junit4]   2> 267819 T1233 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 267819 T1262 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 267820 T1264 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@80c65e name:ZooKeeperConnection Watcher:127.0.0.1:36325/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 267821 T1233 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 267822 T1233 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 267824 T1233 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 267826 T1262 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 267828 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 267829 T1262 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 267829 T1262 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 267829 T1256 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C216 name=collection1 org.apache.solr.core.SolrCore@169e71b url=http://127.0.0.1:43786/collection1 node=127.0.0.1:43786_ C216_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:43786, core=collection1, node_name=127.0.0.1:43786_}
   [junit4]   2> 267829 T1262 C216 P43786 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:43786/collection1/
   [junit4]   2> 267830 T1262 C216 P43786 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 267830 T1262 C216 P43786 oasc.SyncStrategy.syncToMe http://127.0.0.1:43786/collection1/ has no replicas
   [junit4]   2> 267830 T1262 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:43786/collection1/ shard1
   [junit4]   2> 267831 T1262 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 267833 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 267835 T1255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 267835 T1264 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 267839 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 267842 T1256 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 267846 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 267899 T1233 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 267900 T1233 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 267902 T1233 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52358
   [junit4]   2> 267903 T1233 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 267903 T1233 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 267903 T1233 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392636923827
   [junit4]   2> 267904 T1233 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392636923827/'
   [junit4]   2> 267921 T1233 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392636923827/solr.xml
   [junit4]   2> 267949 T1264 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 267949 T1255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 267984 T1233 oasc.CoreContainer.<init> New CoreContainer 4854727
   [junit4]   2> 267984 T1233 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392636923827/]
   [junit4]   2> 267986 T1233 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 267987 T1233 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 267987 T1233 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 267987 T1233 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 267988 T1233 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 267988 T1233 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 267989 T1233 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 267989 T1233 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 267990 T1233 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 267991 T1262 oasc.ZkController.register We are http://127.0.0.1:43786/collection1/ and leader is http://127.0.0.1:43786/collection1/
   [junit4]   2> 267991 T1262 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:43786
   [junit4]   2> 267991 T1262 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 267991 T1262 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 267992 T1262 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 267992 T1233 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 267993 T1233 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 267993 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 267993 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 267993 T1233 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 267993 T1262 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 267994 T1233 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:36325/solr
   [junit4]   2> 267993 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 267995 T1233 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 267995 T1256 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 267995 T1233 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 267996 T1256 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:43786",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:43786_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 267997 T1275 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c6d6d4 name:ZooKeeperConnection Watcher:127.0.0.1:36325 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 267997 T1233 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 267999 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 267999 T1233 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 268000 T1277 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ac72dd name:ZooKeeperConnection Watcher:127.0.0.1:36325/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 268001 T1233 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 268004 T1233 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 268103 T1277 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 268103 T1255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 268103 T1264 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 269007 T1233 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52358_
   [junit4]   2> 269008 T1233 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52358_
   [junit4]   2> 269010 T1264 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 269010 T1255 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 269010 T1277 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 269017 T1278 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 269017 T1278 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 269018 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 269018 T1278 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 269018 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 269019 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 269020 T1256 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 269021 T1256 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52358",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52358_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 269021 T1256 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 269021 T1256 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 269023 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 269126 T1255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 269126 T1264 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 269126 T1277 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 270019 T1278 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 270019 T1278 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392636923827/collection1
   [junit4]   2> 270019 T1278 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 270020 T1278 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 270021 T1278 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 270022 T1278 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 270022 T1278 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392636923827/collection1/'
   [junit4]   2> 270023 T1278 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392636923827/collection1/lib/classes/' to classloader
   [junit4]   2> 270023 T1278 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392636923827/collection1/lib/README' to classloader
   [junit4]   2> 270066 T1278 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 270109 T1278 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 270110 T1278 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 270118 T1278 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 270470 T1278 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 270473 T1278 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 270475 T1278 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 270478 T1278 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 270493 T1278 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 270494 T1278 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392636923827/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392636921895/jetty1/
   [junit4]   2> 270494 T1278 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@16edd1d
   [junit4]   2> 270494 T1278 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392636921895/jetty1
   [junit4]   2> 270495 T1278 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392636921895/jetty1/index/
   [junit4]   2> 270495 T1278 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392636921895/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 270495 T1278 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392636921895/jetty1/index
   [junit4]   2> 270496 T1278 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=26, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 270498 T1278 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392636921895/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 270498 T1278 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 270501 T1278 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 270501 T1278 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 270502 T1278 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 270502 T1278 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 270502 T1278 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 270502 T1278 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 270503 T1278 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 270503 T1278 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 270503 T1278 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 270504 T1278 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 270504 T1278 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 270504 T1278 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 270505 T1278 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 270505 T1278 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 270505 T1278 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 270506 T1278 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 270507 T1278 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 270509 T1278 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 270511 T1278 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 270511 T1278 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 270512 T1278 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=16, maxMergeAtOnceExplicit=14, maxMergedSegmentMB=8.0078125, floorSegmentMB=1.0859375, forceMergeDeletesPctAllowed=21.099702353805615, segmentsPerTier=36.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.79896787184218
   [junit4]   2> 270513 T1278 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392636921895/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 270513 T1278 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 270514 T1278 oass.SolrIndexSearcher.<init> Opening Searcher@87623d[collection1] main
   [junit4]   2> 270518 T1279 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@87623d[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 270524 T1278 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 270525 T1233 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 270526 T1233 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 270525 T1282 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52358 collection:collection1 shard:shard1
   [junit4]   2> 270528 T1282 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 270533 T1282 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 270535 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 270535 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 270535 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 270535 T1282 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 270535 T1282 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C217 name=collection1 org.apache.solr.core.SolrCore@560cae url=http://127.0.0.1:52358/collection1 node=127.0.0.1:52358_ C217_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:52358, core=collection1, node_name=127.0.0.1:52358_}
   [junit4]   2> 270535 T1282 C217 P52358 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52358/collection1/
   [junit4]   2> 270536 T1282 C217 P52358 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 270536 T1282 C217 P52358 oasc.SyncStrategy.syncToMe http://127.0.0.1:52358/collection1/ has no replicas
   [junit4]   2> 270536 T1282 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:52358/collection1/ shard1
   [junit4]   2> 270536 T1282 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 270536 T1256 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 270541 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 270543 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 270548 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 270608 T1233 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 270609 T1233 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 270611 T1233 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60216
   [junit4]   2> 270612 T1233 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 270613 T1233 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 270613 T1233 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392636926529
   [junit4]   2> 270613 T1233 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392636926529/'
   [junit4]   2> 270625 T1233 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392636926529/solr.xml
   [junit4]   2> 270651 T1264 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 270651 T1277 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 270651 T1255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 270666 T1233 oasc.CoreContainer.<init> New CoreContainer 4488117
   [junit4]   2> 270667 T1233 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392636926529/]
   [junit4]   2> 270668 T1233 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 270668 T1233 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 270668 T1233 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 270669 T1233 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 270669 T1233 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 270669 T1233 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 270670 T1233 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 270670 T1233 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 270670 T1233 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 270672 T1233 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 270673 T1233 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 270673 T1233 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 270673 T1233 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:36325/solr
   [junit4]   2> 270674 T1233 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 270674 T1233 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 270676 T1293 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15b06b name:ZooKeeperConnection Watcher:127.0.0.1:36325 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 270676 T1233 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 270678 T1233 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 270679 T1295 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6beb8b name:ZooKeeperConnection Watcher:127.0.0.1:36325/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 270680 T1233 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 270683 T1233 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 270694 T1282 oasc.ZkController.register We are http://127.0.0.1:52358/collection1/ and leader is http://127.0.0.1:52358/collection1/
   [junit4]   2> 270694 T1282 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52358
   [junit4]   2> 270694 T1282 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 270695 T1282 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 270695 T1282 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 270696 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 270696 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 270696 T1282 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 270696 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 270697 T1256 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 270698 T1256 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52358",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52358_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 270700 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 270802 T1264 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 270802 T1295 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 270802 T1277 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 270802 T1255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 271685 T1233 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60216_
   [junit4]   2> 271686 T1233 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60216_
   [junit4]   2> 271689 T1264 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 271689 T1295 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 271689 T1277 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 271689 T1255 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 271695 T1296 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 271695 T1296 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 271697 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 271697 T1296 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 271697 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 271698 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 271699 T1256 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 271700 T1256 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:60216",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:60216_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 271700 T1256 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 271700 T1256 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 271702 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 271805 T1255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 271805 T1277 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 271805 T1295 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 271805 T1264 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 272697 T1296 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 272698 T1296 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392636926529/collection1
   [junit4]   2> 272698 T1296 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 272699 T1296 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 272699 T1296 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 272700 T1296 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 272700 T1296 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392636926529/collection1/'
   [junit4]   2> 272701 T1296 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392636926529/collection1/lib/classes/' to classloader
   [junit4]   2> 272701 T1296 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392636926529/collection1/lib/README' to classloader
   [junit4]   2> 272739 T1296 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 272787 T1296 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 272789 T1296 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 272796 T1296 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 273152 T1296 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 273155 T1296 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 273157 T1296 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 273161 T1296 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 273181 T1296 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 273181 T1296 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392636926529/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392636921895/jetty2/
   [junit4]   2> 273181 T1296 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@16edd1d
   [junit4]   2> 273182 T1296 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392636921895/jetty2
   [junit4]   2> 273182 T1296 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392636921895/jetty2/index/
   [junit4]   2> 273183 T1296 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392636921895/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 273183 T1296 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392636921895/jetty2/index
   [junit4]   2> 273184 T1296 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=26, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 273186 T1296 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392636921895/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 273186 T1296 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 273189 T1296 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 273190 T1296 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 273190 T1296 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 273190 T1296 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 273190 T1296 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 273191 T1296 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 273191 T1296 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 273191 T1296 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 273191 T1296 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 273192 T1296 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 273193 T1296 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 273193 T1296 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 273193 T1296 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 273194 T1296 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 273194 T1296 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 273195 T1296 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 273195 T1296 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 273197 T1296 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 273200 T1296 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 273200 T1296 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 273201 T1296 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=16, maxMergeAtOnceExplicit=14, maxMergedSegmentMB=8.0078125, floorSegmentMB=1.0859375, forceMergeDeletesPctAllowed=21.099702353805615, segmentsPerTier=36.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.79896787184218
   [junit4]   2> 273202 T1296 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392636921895/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 273202 T1296 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 273202 T1296 oass.SolrIndexSearcher.<init> Opening Searcher@10c715e[collection1] main
   [junit4]   2> 273206 T1297 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@10c715e[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 273210 T1296 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 273210 T1233 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 273211 T1233 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 273210 T1300 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:60216 collection:collection1 shard:shard2
   [junit4]   2> 273212 T1300 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 273216 T1300 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 273218 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 273218 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 273218 T1300 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 273218 T1300 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 273218 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 273219 T1256 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C218 name=collection1 org.apache.solr.core.SolrCore@be8a14 url=http://127.0.0.1:60216/collection1 node=127.0.0.1:60216_ C218_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:60216, core=collection1, node_name=127.0.0.1:60216_}
   [junit4]   2> 273219 T1300 C218 P60216 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:60216/collection1/
   [junit4]   2> 273219 T1300 C218 P60216 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 273220 T1300 C218 P60216 oasc.SyncStrategy.syncToMe http://127.0.0.1:60216/collection1/ has no replicas
   [junit4]   2> 273220 T1300 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:60216/collection1/ shard2
   [junit4]   2> 273220 T1300 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 273221 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 273224 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 273226 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 273279 T1233 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 273279 T1233 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 273281 T1233 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:42986
   [junit4]   2> 273282 T1233 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 273283 T1233 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 273283 T1233 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1392636929213
   [junit4]   2> 273283 T1233 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1392636929213/'
   [junit4]   2> 273294 T1233 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1392636929213/solr.xml
   [junit4]   2> 273331 T1264 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 273331 T1277 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 273331 T1295 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 273331 T1255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 273336 T1233 oasc.CoreContainer.<init> New CoreContainer 11184025
   [junit4]   2> 273337 T1233 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1392636929213/]
   [junit4]   2> 273338 T1233 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 273338 T1233 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 273339 T1233 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 273339 T1233 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 273339 T1233 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 273340 T1233 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 273340 T1233 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 273340 T1233 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 273340 T1233 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 273343 T1233 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 273343 T1233 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 273343 T1233 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 273344 T1233 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:36325/solr
   [junit4]   2> 273344 T1233 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 273345 T1233 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 273346 T1311 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e142aa name:ZooKeeperConnection Watcher:127.0.0.1:36325 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 273346 T1233 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 273348 T1233 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 273349 T1313 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@94dcd0 name:ZooKeeperConnection Watcher:127.0.0.1:36325/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 273350 T1233 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 273352 T1233 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 273374 T1300 oasc.ZkController.register We are http://127.0.0.1:60216/collection1/ and leader is http://127.0.0.1:60216/collection1/
   [junit4]   2> 273375 T1300 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:60216
   [junit4]   2> 273375 T1300 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 273375 T1300 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 273375 T1300 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 273376 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 273376 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 273377 T1300 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 273377 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 273378 T1256 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 273378 T1256 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:60216",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:60216_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 273380 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 273482 T1255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 273482 T1295 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 273482 T1277 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 273482 T1264 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 273483 T1313 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 274354 T1233 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42986_
   [junit4]   2> 274356 T1233 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42986_
   [junit4]   2> 274358 T1264 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 274359 T1277 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 274358 T1295 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 274359 T1255 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 274359 T1313 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 274365 T1314 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 274365 T1314 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 274367 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 274367 T1314 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 274367 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 274367 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 274369 T1256 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 274370 T1256 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:42986",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:42986_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 274370 T1256 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 274370 T1256 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 274373 T1255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 274476 T1313 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 274476 T1255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 274476 T1264 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 274476 T1277 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 274476 T1295 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 275367 T1314 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 275368 T1314 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1392636929213/collection1
   [junit4]   2> 275368 T1314 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 275368 T1314 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 275369 T1314 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 275370 T1314 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 275370 T1314 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1392636929213/collection1/'
   [junit4]   2> 275370 T1314 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1392636929213/collection1/lib/classes/' to classloader
   [junit4]   2> 275371 T1314 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1392636929213/collection1/lib/README' to classloader
   [junit4]   2> 275407 T1314 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 275456 T1314 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 275457 T1314 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 275465 T1314 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 275836 T1314 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 275838 T1314 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 275840 T1314 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 275843 T1314 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 275859 T1314 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 275859 T1314 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1392636929213/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392636921895/jetty3/
   [junit4]   2> 275859 T1314 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@16edd1d
   [junit4]   2> 275860 T1314 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392636921895/jetty3
   [junit4]   2> 275860 T1314 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392636921895/jetty3/index/
   [junit4]   2> 275860 T1314 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392636921895/jetty3/index' doesn't exist. Creating new index...
   [junit4]   2> 275861 T1314 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392636921895/jetty3/index
   [junit4]   2> 275861 T1314 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePoli

[...truncated too long message...]

 1>    /solr/live_nodes/127.0.0.1:60216_ (0)
   [junit4]   1>    /solr/live_nodes/127.0.0.1:33380_ (0)
   [junit4]   1>    /solr/live_nodes/127.0.0.1:60589_ (0)
   [junit4]   1>    /solr/live_nodes/127.0.0.1:32780_ (0)
   [junit4]   1>    /solr/live_nodes/127.0.0.1:43786_ (0)
   [junit4]   1>    /solr/live_nodes/127.0.0.1:42986_ (0)
   [junit4]   1>    /solr/live_nodes/127.0.0.1:33305_ (0)
   [junit4]   1>   /solr/configs (1)
   [junit4]   1>    /solr/configs/conf1 (11)
   [junit4]   1>     /solr/configs/conf1/solrconfig.xml (0)
   [junit4]   1>     DATA: ...supressed...
   [junit4]   1>     /solr/configs/conf1/synonyms.txt (0)
   [junit4]   1>     DATA: ...supressed...
   [junit4]   1>     /solr/configs/conf1/open-exchange-rates.json (0)
   [junit4]   1>     DATA:
   [junit4]   1>         {
   [junit4]   1>           "disclaimer": "This data is not real, it was synthetically created to match currency.xml.  It is modeled after the data format available from openexchangerates.org.  See https://openexchangerates.org/documentation for details",
   [junit4]   1>           "license": "http://www.apache.org/licenses/LICENSE-2.0",
   [junit4]   1>           "timestamp": 1332070464,
   [junit4]   1>         
   [junit4]   1>         
   [junit4]   1>           "IMPORTANT NOTE": "In order for tests to work, this data must be kept in sync with ./currency.xml",
   [junit4]   1>         
   [junit4]   1>         
   [junit4]   1>           "base": "USD",
   [junit4]   1>           "rates": {
   [junit4]   1>             "USD": 1,
   [junit4]   1>             "JPY": 81.29,
   [junit4]   1>             "EUR": 2.5,
   [junit4]   1>             "GBP": 0.5,
   [junit4]   1>             "MXN": 2.0
   [junit4]   1>           }
   [junit4]   1>         }
   [junit4]   1>         
   [junit4]   1>     /solr/configs/conf1/currency.xml (0)
   [junit4]   1>     DATA: ...supressed...
   [junit4]   1>     /solr/configs/conf1/schema.xml (0)
   [junit4]   1>     DATA: ...supressed...
   [junit4]   1>     /solr/configs/conf1/mapping-ISOLatin1Accent.txt (0)
   [junit4]   1>     DATA: ...supressed...
   [junit4]   1>     /solr/configs/conf1/enumsConfig.xml (0)
   [junit4]   1>     DATA: ...supressed...
   [junit4]   1>     /solr/configs/conf1/stopwords.txt (0)
   [junit4]   1>     DATA: ...supressed...
   [junit4]   1>     /solr/configs/conf1/protwords.txt (0)
   [junit4]   1>     DATA: ...supressed...
   [junit4]   1>     /solr/configs/conf1/old_synonyms.txt (0)
   [junit4]   1>     DATA: ...supressed...
   [junit4]   1>     /solr/configs/conf1/solrconfig.snippet.randomindexconfig.xml (0)
   [junit4]   1>     DATA: ...supressed...
   [junit4]   1>  /zookeeper (1)
   [junit4]   1>  DATA:
   [junit4]   1>      
   [junit4]   1> 
   [junit4]   2> 546021 T1818 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 546027 T1818 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 546028 T1818 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 546028 T1818 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392636921895/jetty7/index.20140217213932660 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392636921895/jetty7/index.20140217213932660;done=false>>]
   [junit4]   2> 546028 T1818 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392636921895/jetty7/index.20140217213932660
   [junit4]   2> 546028 T1818 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392636921895/jetty7 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392636921895/jetty7;done=false>>]
   [junit4]   2> 546029 T1818 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392636921895/jetty7
   [junit4]   2> 546029 T1818 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 546029 T1818 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 546030 T1818 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 546030 T1818 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ChaosMonkeyNothingIsSafeTest -Dtests.method=testDistribSearch -Dtests.seed=AB9C3D0931F47C85 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_SY -Dtests.timezone=Pacific/Saipan -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE  280s J1 | ChaosMonkeyNothingIsSafeTest.testDistribSearch <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: document count mismatch.  control=1952 sum(shards)=1951 cloudClient=1951
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([AB9C3D0931F47C85:2A7AB31146AB1CB9]:0)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1230)
   [junit4]    > 	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:213)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 546042 T1233 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Lucene42, sim=DefaultSimilarity, locale=ar_SY, timezone=Pacific/Saipan
   [junit4]   2> NOTE: Linux 3.8.0-35-generic i386/Oracle Corporation 1.7.0_51 (32-bit)/cpus=8,threads=1,free=99068736,total=267124736
   [junit4]   2> NOTE: All tests run in this JVM: [TestCoreDiscovery, TestOmitPositions, TestSchemaSimilarityResource, StandardRequestHandlerTest, TestFastLRUCache, TestCloudManagedSchemaAddField, TestSolrJ, NoCacheHeaderTest, TestMultiCoreConfBootstrap, TestUniqueKeyFieldResource, TestComponentsName, ZkSolrClientTest, TestModifyConfFiles, TestFastWriter, DebugComponentTest, TestSimpleQParserPlugin, JSONWriterTest, TestBlendedInfixSuggestions, UniqFieldsUpdateProcessorFactoryTest, FileUtilsTest, DisMaxRequestHandlerTest, TestNonDefinedSimilarityFactory, TestDistributedSearch, TestUpdate, TestCodecSupport, TestFieldTypeCollectionResource, TestIndexSearcher, DocValuesMultiTest, HighlighterConfigTest, SolrTestCaseJ4Test, HdfsBasicDistributedZkTest, FullSolrCloudDistribCmdsTest, ConvertedLegacyTest, TestZkChroot, FieldAnalysisRequestHandlerTest, TestSolrXmlPersistence, ChaosMonkeyNothingIsSafeTest]
   [junit4] Completed on J1 in 280.18s, 1 test, 1 failure <<< FAILURES!

[...truncated 873 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:453: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:433: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1268: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:901: There were test failures: 372 suites, 1604 tests, 1 failure, 35 ignored (7 assumptions)

Total time: 55 minutes 31 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_51 -client -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0-fcs-b129) - Build # 9504 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9504/
Java: 64bit/jdk1.8.0-fcs-b129 -XX:-UseCompressedOops -XX:+UseSerialGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.testDistribSearch

Error Message:
document count mismatch.  control=1778 sum(shards)=1777 cloudClient=1777

Stack Trace:
java.lang.AssertionError: document count mismatch.  control=1778 sum(shards)=1777 cloudClient=1777
	at __randomizedtesting.SeedInfo.seed([252503FF34E84BC4:A4C38DE743B72BF8]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1230)
	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:213)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 11045 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
   [junit4]   2> 1177258 T5712 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 1177259 T5712 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /cse/cm
   [junit4]   2> 1177263 T5712 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeyNothingIsSafeTest-1392624903263
   [junit4]   2> 1177263 T5712 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1177264 T5713 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1177364 T5712 oasc.ZkTestServer.run start zk server on port:58224
   [junit4]   2> 1177365 T5712 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1177463 T5719 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2fe19f94 name:ZooKeeperConnection Watcher:127.0.0.1:58224 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1177464 T5712 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1177464 T5712 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1177468 T5712 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1177472 T5721 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@204ae5d8 name:ZooKeeperConnection Watcher:127.0.0.1:58224/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1177472 T5712 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1177473 T5712 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1177476 T5712 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1177478 T5712 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1177481 T5712 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1177483 T5712 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1177484 T5712 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1177487 T5712 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 1177488 T5712 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1177495 T5712 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1177495 T5712 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1177499 T5712 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1177500 T5712 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1177505 T5712 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1177505 T5712 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1177508 T5712 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1177509 T5712 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1177511 T5712 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1177512 T5712 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1177514 T5712 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1177515 T5712 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1177518 T5712 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1177519 T5712 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1177521 T5712 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1177522 T5712 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1177524 T5712 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1177525 T5712 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1177536 T5712 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1177540 T5723 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7cda08df name:ZooKeeperConnection Watcher:127.0.0.1:58224/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1177540 T5712 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1177654 T5712 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1177671 T5712 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1177681 T5712 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:46659
   [junit4]   2> 1177683 T5712 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1177683 T5712 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1177684 T5712 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392624903543
   [junit4]   2> 1177684 T5712 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392624903543/'
   [junit4]   2> 1177704 T5712 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392624903543/solr.xml
   [junit4]   2> 1177716 T5712 oasc.CoreContainer.<init> New CoreContainer 1134510240
   [junit4]   2> 1177717 T5712 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392624903543/]
   [junit4]   2> 1177717 T5712 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1177718 T5712 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1177718 T5712 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1177719 T5712 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1177719 T5712 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1177719 T5712 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1177720 T5712 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1177720 T5712 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1177721 T5712 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1177725 T5712 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1177726 T5712 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1177726 T5712 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1177727 T5712 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58224/solr
   [junit4]   2> 1177727 T5712 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1177728 T5712 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1177730 T5734 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7bc82e71 name:ZooKeeperConnection Watcher:127.0.0.1:58224 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1177732 T5712 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1177737 T5712 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1177738 T5736 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@607a67df name:ZooKeeperConnection Watcher:127.0.0.1:58224/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1177738 T5712 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1177740 T5712 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1177743 T5712 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1177746 T5712 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1177748 T5712 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46659_cse%2Fcm
   [junit4]   2> 1177749 T5712 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46659_cse%2Fcm
   [junit4]   2> 1177752 T5712 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1177754 T5712 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1177758 T5712 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:46659_cse%2Fcm
   [junit4]   2> 1177758 T5712 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1177760 T5712 oasc.Overseer.start Overseer (id=91267065660375044-127.0.0.1:46659_cse%2Fcm-n_0000000000) starting
   [junit4]   2> 1177763 T5712 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1177767 T5738 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1177767 T5712 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1177768 T5738 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 1177771 T5712 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1177774 T5712 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1177777 T5737 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1177777 T5739 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1177777 T5739 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1177778 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1177778 T5739 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1177779 T5737 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1177780 T5737 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:46659/cse/cm",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:46659_cse%2Fcm",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1177780 T5737 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 1177780 T5737 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1177782 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1177782 T5736 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1178779 T5739 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1178779 T5739 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392624903543/collection1
   [junit4]   2> 1178779 T5739 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 1178780 T5739 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1178780 T5739 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1178781 T5739 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1178781 T5739 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392624903543/collection1/'
   [junit4]   2> 1178782 T5739 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392624903543/collection1/lib/classes/' to classloader
   [junit4]   2> 1178782 T5739 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392624903543/collection1/lib/README' to classloader
   [junit4]   2> 1178814 T5739 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1178825 T5739 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1178826 T5739 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1178830 T5739 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1178934 T5739 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1178935 T5739 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1178936 T5739 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1178938 T5739 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1178950 T5739 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1178951 T5739 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392624903543/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392624903263/control/data/
   [junit4]   2> 1178951 T5739 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@354acb04
   [junit4]   2> 1178951 T5739 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392624903263/control/data
   [junit4]   2> 1178952 T5739 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392624903263/control/data/index/
   [junit4]   2> 1178952 T5739 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392624903263/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1178952 T5739 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392624903263/control/data/index
   [junit4]   2> 1178952 T5739 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=33.595703125, floorSegmentMB=0.2451171875, forceMergeDeletesPctAllowed=29.798192879937254, segmentsPerTier=18.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1179006 T5739 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392624903263/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 1179007 T5739 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1179008 T5739 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1179009 T5739 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1179009 T5739 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1179009 T5739 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1179009 T5739 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1179009 T5739 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1179010 T5739 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1179010 T5739 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1179010 T5739 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1179010 T5739 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1179010 T5739 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1179010 T5739 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1179011 T5739 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1179011 T5739 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1179011 T5739 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1179011 T5739 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1179012 T5739 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1179013 T5739 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1179014 T5739 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1179014 T5739 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1179015 T5739 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=47, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=32.3115234375, floorSegmentMB=1.9111328125, forceMergeDeletesPctAllowed=24.60254598273686, segmentsPerTier=33.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1179015 T5739 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392624903263/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 1179015 T5739 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1179016 T5739 oass.SolrIndexSearcher.<init> Opening Searcher@107f2667[collection1] main
   [junit4]   2> 1179017 T5740 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@107f2667[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1179017 T5739 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1179018 T5712 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1179018 T5712 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1179018 T5743 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:46659/cse/cm collection:control_collection shard:shard1
   [junit4]   2> 1179020 T5743 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1179020 T5712 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1179021 T5745 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e6d7469 name:ZooKeeperConnection Watcher:127.0.0.1:58224/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1179022 T5712 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1179022 T5712 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1179024 T5712 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1179025 T5743 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1179026 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1179026 T5743 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1179027 T5743 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C4158 name=collection1 org.apache.solr.core.SolrCore@169a26d9 url=https://127.0.0.1:46659/cse/cm/collection1 node=127.0.0.1:46659_cse%2Fcm C4158_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:46659/cse/cm, core=collection1, node_name=127.0.0.1:46659_cse%2Fcm}
   [junit4]   2> 1179027 T5743 C4158 P46659 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:46659/cse/cm/collection1/
   [junit4]   2> 1179027 T5743 C4158 P46659 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1179027 T5737 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1179027 T5743 C4158 P46659 oasc.SyncStrategy.syncToMe https://127.0.0.1:46659/cse/cm/collection1/ has no replicas
   [junit4]   2> 1179027 T5743 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:46659/cse/cm/collection1/ shard1
   [junit4]   2> 1179027 T5743 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1179029 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1179031 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1179033 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1179089 T5712 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 1179090 T5712 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1179093 T5712 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1179094 T5712 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:36026
   [junit4]   2> 1179096 T5712 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1179096 T5712 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1179096 T5712 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392624905024
   [junit4]   2> 1179096 T5712 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392624905024/'
   [junit4]   2> 1179107 T5712 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392624905024/solr.xml
   [junit4]   2> 1179115 T5712 oasc.CoreContainer.<init> New CoreContainer 1915955316
   [junit4]   2> 1179115 T5712 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392624905024/]
   [junit4]   2> 1179116 T5712 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1179116 T5712 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1179116 T5712 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1179116 T5712 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1179116 T5712 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1179117 T5712 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1179117 T5712 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1179117 T5712 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1179117 T5712 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1179120 T5712 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1179120 T5712 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1179121 T5712 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1179121 T5712 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58224/solr
   [junit4]   2> 1179121 T5712 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1179122 T5712 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1179124 T5756 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@65e8329f name:ZooKeeperConnection Watcher:127.0.0.1:58224 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1179124 T5712 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1179126 T5712 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1179127 T5758 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@55a7028 name:ZooKeeperConnection Watcher:127.0.0.1:58224/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1179127 T5712 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1179130 T5712 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1179135 T5736 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1179135 T5758 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1179135 T5745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1179182 T5743 oasc.ZkController.register We are https://127.0.0.1:46659/cse/cm/collection1/ and leader is https://127.0.0.1:46659/cse/cm/collection1/
   [junit4]   2> 1179182 T5743 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:46659/cse/cm
   [junit4]   2> 1179182 T5743 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1179182 T5743 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 1179183 T5743 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1179183 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1179184 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1179184 T5743 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1179184 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1179185 T5737 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1179186 T5737 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:46659/cse/cm",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:46659_cse%2Fcm",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1179187 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1179289 T5745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1179289 T5758 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1179289 T5736 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1180132 T5712 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36026_cse%2Fcm
   [junit4]   2> 1180134 T5712 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36026_cse%2Fcm
   [junit4]   2> 1180136 T5745 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1180136 T5736 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1180136 T5758 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1180151 T5759 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1180151 T5759 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1180152 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1180152 T5759 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1180152 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1180153 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1180153 T5737 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1180154 T5737 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:36026/cse/cm",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:36026_cse%2Fcm",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1180154 T5737 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 1180154 T5737 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 1180156 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1180258 T5736 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1180258 T5758 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1180258 T5745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1181153 T5759 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1181153 T5759 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392624905024/collection1
   [junit4]   2> 1181153 T5759 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1181154 T5759 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1181154 T5759 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1181155 T5759 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1181155 T5759 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392624905024/collection1/'
   [junit4]   2> 1181155 T5759 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392624905024/collection1/lib/classes/' to classloader
   [junit4]   2> 1181156 T5759 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392624905024/collection1/lib/README' to classloader
   [junit4]   2> 1181180 T5759 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1181192 T5759 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1181193 T5759 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1181197 T5759 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1181316 T5759 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1181318 T5759 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1181318 T5759 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1181320 T5759 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1181334 T5759 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1181334 T5759 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392624905024/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392624903263/jetty1/
   [junit4]   2> 1181334 T5759 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@354acb04
   [junit4]   2> 1181335 T5759 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392624903263/jetty1
   [junit4]   2> 1181335 T5759 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392624903263/jetty1/index/
   [junit4]   2> 1181335 T5759 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392624903263/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 1181336 T5759 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392624903263/jetty1/index
   [junit4]   2> 1181336 T5759 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=33.595703125, floorSegmentMB=0.2451171875, forceMergeDeletesPctAllowed=29.798192879937254, segmentsPerTier=18.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1181340 T5759 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392624903263/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 1181340 T5759 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1181342 T5759 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1181343 T5759 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1181344 T5759 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1181344 T5759 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1181344 T5759 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1181344 T5759 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1181345 T5759 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1181345 T5759 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1181345 T5759 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1181346 T5759 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1181346 T5759 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1181346 T5759 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1181347 T5759 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1181347 T5759 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1181348 T5759 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1181348 T5759 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1181349 T5759 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1181351 T5759 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1181352 T5759 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1181352 T5759 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1181353 T5759 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=47, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=32.3115234375, floorSegmentMB=1.9111328125, forceMergeDeletesPctAllowed=24.60254598273686, segmentsPerTier=33.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1181354 T5759 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392624903263/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 1181355 T5759 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1181355 T5759 oass.SolrIndexSearcher.<init> Opening Searcher@487e1f06[collection1] main
   [junit4]   2> 1181359 T5759 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1181359 T5712 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1181360 T5712 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1181359 T5763 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:36026/cse/cm collection:collection1 shard:shard2
   [junit4]   2> 1181359 T5760 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@487e1f06[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1181365 T5763 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 1181369 T5763 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 1181371 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1181371 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1181371 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1181371 T5763 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1181372 T5763 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C4159 name=collection1 org.apache.solr.core.SolrCore@2cac4099 url=https://127.0.0.1:36026/cse/cm/collection1 node=127.0.0.1:36026_cse%2Fcm C4159_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:36026/cse/cm, core=collection1, node_name=127.0.0.1:36026_cse%2Fcm}
   [junit4]   2> 1181372 T5763 C4159 P36026 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:36026/cse/cm/collection1/
   [junit4]   2> 1181372 T5737 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1181372 T5763 C4159 P36026 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1181373 T5763 C4159 P36026 oasc.SyncStrategy.syncToMe https://127.0.0.1:36026/cse/cm/collection1/ has no replicas
   [junit4]   2> 1181373 T5763 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:36026/cse/cm/collection1/ shard2
   [junit4]   2> 1181373 T5763 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 1181374 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1181378 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1181381 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1181456 T5712 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 1181456 T5712 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1181458 T5712 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1181460 T5712 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:43378
   [junit4]   2> 1181461 T5712 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1181462 T5712 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1181462 T5712 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392624907361
   [junit4]   2> 1181462 T5712 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392624907361/'
   [junit4]   2> 1181473 T5712 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392624907361/solr.xml
   [junit4]   2> 1181494 T5745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1181494 T5758 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1181494 T5736 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1181495 T5712 oasc.CoreContainer.<init> New CoreContainer 718403903
   [junit4]   2> 1181495 T5712 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392624907361/]
   [junit4]   2> 1181495 T5712 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1181496 T5712 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1181496 T5712 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1181496 T5712 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1181497 T5712 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1181497 T5712 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1181497 T5712 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1181497 T5712 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1181498 T5712 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1181500 T5712 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1181501 T5712 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1181501 T5712 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1181501 T5712 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58224/solr
   [junit4]   2> 1181501 T5712 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1181502 T5712 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1181504 T5774 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ce2edc7 name:ZooKeeperConnection Watcher:127.0.0.1:58224 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1181504 T5712 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1181506 T5712 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1181507 T5776 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@62444952 name:ZooKeeperConnection Watcher:127.0.0.1:58224/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1181507 T5712 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1181510 T5712 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1181542 T5763 oasc.ZkController.register We are https://127.0.0.1:36026/cse/cm/collection1/ and leader is https://127.0.0.1:36026/cse/cm/collection1/
   [junit4]   2> 1181542 T5763 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:36026/cse/cm
   [junit4]   2> 1181542 T5763 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1181542 T5763 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 1181542 T5763 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1181544 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1181544 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1181544 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1181544 T5763 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1181546 T5737 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1181546 T5737 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:36026/cse/cm",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:36026_cse%2Fcm",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1181548 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1181650 T5745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1181650 T5736 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1181650 T5776 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1181650 T5758 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1182513 T5712 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43378_cse%2Fcm
   [junit4]   2> 1182514 T5712 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43378_cse%2Fcm
   [junit4]   2> 1182517 T5745 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1182517 T5758 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1182518 T5736 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1182518 T5776 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1182521 T5777 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1182521 T5777 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1182522 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1182522 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1182522 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1182522 T5777 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1182523 T5737 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1182524 T5737 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:43378/cse/cm",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:43378_cse%2Fcm",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1182524 T5737 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 1182525 T5737 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1182526 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1182628 T5758 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1182628 T5776 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1182628 T5736 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1182628 T5745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1183523 T5777 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1183523 T5777 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392624907361/collection1
   [junit4]   2> 1183524 T5777 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1183524 T5777 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1183524 T5777 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1183525 T5777 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1183525 T5777 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392624907361/collection1/'
   [junit4]   2> 1183526 T5777 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392624907361/collection1/lib/classes/' to classloader
   [junit4]   2> 1183526 T5777 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392624907361/collection1/lib/README' to classloader
   [junit4]   2> 1183548 T5777 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1183558 T5777 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1183560 T5777 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1183565 T5777 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1183704 T5777 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1183706 T5777 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1183707 T5777 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1183709 T5777 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1183728 T5777 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1183729 T5777 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392624907361/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392624903263/jetty2/
   [junit4]   2> 1183729 T5777 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@354acb04
   [junit4]   2> 1183729 T5777 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392624903263/jetty2
   [junit4]   2> 1183730 T5777 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392624903263/jetty2/index/
   [junit4]   2> 1183730 T5777 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392624903263/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 1183730 T5777 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392624903263/jetty2/index
   [junit4]   2> 1183731 T5777 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=33.595703125, floorSegmentMB=0.2451171875, forceMergeDeletesPctAllowed=29.798192879937254, segmentsPerTier=18.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1183734 T5777 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392624903263/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 1183735 T5777 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1183738 T5777 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1183738 T5777 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1183739 T5777 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1183739 T5777 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1183739 T5777 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1183739 T5777 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1183740 T5777 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1183740 T5777 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1183740 T5777 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1183741 T5777 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1183741 T5777 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1183741 T5777 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1183742 T5777 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1183742 T5777 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1183742 T5777 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1183743 T5777 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1183743 T5777 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1183745 T5777 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1183747 T5777 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1183747 T5777 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1183748 T5777 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=47, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=32.3115234375, floorSegmentMB=1.9111328125, forceMergeDeletesPctAllowed=24.60254598273686, segmentsPerTier=33.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1183749 T5777 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392624903263/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 1183749 T5777 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1183749 T5777 oass.SolrIndexSearcher.<init> Opening Searcher@5a5b3625[collection1] main
   [junit4]   2> 1183751 T5778 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5a5b3625[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1183753 T5777 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1183754 T5712 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1183754 T5712 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1183754 T5781 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:43378/cse/cm collection:collection1 shard:shard1
   [junit4]   2> 1183755 T5781 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1183759 T5781 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1183761 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1183761 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1183761 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1183761 T5781 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1183762 T5781 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C4160 name=collection1 org.apache.solr.core.SolrCore@67299229 url=https://127.0.0.1:43378/cse/cm/collection1 node=127.0.0.1:43378_cse%2Fcm C4160_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:43378/cse/cm, core=collection1, node_name=127.0.0.1:43378_cse%2Fcm}
   [junit4]   2> 1183762 T5781 C4160 P43378 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:43378/cse/cm/collection1/
   [junit4]   2> 1183762 T5737 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1183762 T5781 C4160 P43378 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1183763 T5781 C4160 P43378 oasc.SyncStrategy.syncToMe https://127.0.0.1:43378/cse/cm/collection1/ has no replicas
   [junit4]   2> 1183763 T5781 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:43378/cse/cm/collection1/ shard1
   [junit4]   2> 1183763 T5781 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1183765 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1183768 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1183770 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1183859 T5712 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 1183859 T5712 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1183861 T5712 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1183862 T5712 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:40916
   [junit4]   2> 1183864 T5712 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1183864 T5712 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1183865 T5712 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1392624909756
   [junit4]   2> 1183865 T5712 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1392624909756/'
   [junit4]   2> 1183892 T5758 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1183892 T5736 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1183892 T5745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1183892 T5776 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1183895 T5712 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1392624909756/solr.xml
   [junit4]   2> 1183906 T5712 oasc.CoreContainer.<init> New CoreContainer 1834393288
   [junit4]   2> 1183907 T5712 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1392624909756/]
   [junit4]   2> 1183908 T5712 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1183908 T5712 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1183908 T5712 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1183909 T5712 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1183909 T5712 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1183909 T5712 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1183910 T5712 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1183910 T5712 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1183910 T5712 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1183914 T5712 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1183915 T5712 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1183915 T5712 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1183916 T5712 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58224/solr
   [junit4]   2> 1183916 T5712 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1183918 T5712 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1183919 T5781 oasc.ZkController.register We are https://127.0.0.1:43378/cse/cm/collection1/ and leader is https://127.0.0.1:43378/cse/cm/collection1/
   [junit4]   2> 1183919 T5781 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:43378/cse/cm
   [junit4]   2> 1183919 T5781 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1183920 T5781 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 1183920 T5781 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1183920 T5792 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6636cfdf name:ZooKeeperConnection Watcher:127.0.0.1:58224 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1183920 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1183921 T5712 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1183921 T5781 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1183921 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1183922 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1183922 T5737 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1183923 T5737 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:43378/cse/cm",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:43378_cse%2Fcm",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 1183925 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1183930 T5712 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1183942 T5794 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@24f30324 name:ZooKeeperConnection Watcher:127.0.0.1:58224/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1183943 T5712 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1183946 T5712 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1184026 T5794 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1184026 T5736 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1184026 T5776 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1184026 T5745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1184026 T5758 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1184953 T5712 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40916_cse%2Fcm
   [junit4]   2> 1184954 T5712 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40916_cse%2Fcm
   [junit4]   2> 1184956 T5736 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1184956 T5794 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1184956 T5758 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1184956 T5776 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1184956 T5745 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1184959 T5795 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1184959 T5795 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1184961 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1184961 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1184961 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1184961 T5795 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1184962 T5737 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1184962 T5737 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:40916/cse/cm",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:40916_cse%2Fcm",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1184963 T5737 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 1184963 T5737 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 1184964 T5736 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1185066 T5745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1185066 T5758 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1185067 T5736 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1185070 T5794 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1185070 T5776 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1185962 T5795 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1185962 T5795 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1392624909756/collection1
   [junit4]   2> 1185963 T5795 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1185963 T5795 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1185963 T5795 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1185964 T5795 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1185964 T5795 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1392624909756/collection1/'
   [junit4]   2> 1185965 T5795 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1392624909756/collection1/lib/classes/' to classloader
   [junit4]   2> 1185965 T5795 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1392624909756/collection1/lib/README' to classloader
   [junit4]   2> 1185987 T5795 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1186000 T5795 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1186002 T5795 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1186008 T5795 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4

[...truncated too long message...]

 <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: document count mismatch.  control=1778 sum(shards)=1777 cloudClient=1777
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([252503FF34E84BC4:A4C38DE743B72BF8]:0)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1230)
   [junit4]    > 	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:213)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 1448248 T5712 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 1448708 T5736 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1448708 T5736 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1448796 T6222 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1448894 T5794 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1449089 T6306 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1449911 T5872 C5432 P54406 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
   [junit4]   2> 1449911 T5872 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
   [junit4]   2> 1449912 T5872 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@225765ec
   [junit4]   2> 1449913 T5872 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=2,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=1777,cumulative_deletesById=880,cumulative_deletesByQuery=1,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=1}
   [junit4]   2> 1449913 T5872 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 1449913 T5872 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1449913 T5872 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 1449914 T5872 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1449915 T5872 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1449915 T5872 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392624903263/jetty7 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392624903263/jetty7;done=false>>]
   [junit4]   2> 1449915 T5872 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392624903263/jetty7
   [junit4]   2> 1449915 T5872 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392624903263/jetty7/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392624903263/jetty7/index;done=false>>]
   [junit4]   2> 1449915 T5872 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392624903263/jetty7/index
   [junit4]   2> 1449916 T5872 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1449916 T5872 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1449916 T5872 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1449916 T5872 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1449916 T5872 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=Lucene42, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=ca_ES, timezone=Africa/Niamey
   [junit4]   2> NOTE: Linux 3.8.0-35-generic amd64/Oracle Corporation 1.8.0 (64-bit)/cpus=8,threads=1,free=149371736,total=431091712
   [junit4]   2> NOTE: All tests run in this JVM: [UniqFieldsUpdateProcessorFactoryTest, TestStandardQParsers, RemoteQueryErrorTest, TestAtomicUpdateErrorCases, PrimUtilsTest, SpellCheckCollatorTest, TestClassNameShortening, DirectSolrSpellCheckerTest, CurrencyFieldOpenExchangeTest, CursorPagingTest, PreAnalyzedUpdateProcessorTest, TimeZoneUtilsTest, TestSweetSpotSimilarityFactory, TestRecovery, TestWordDelimiterFilterFactory, TestSchemaNameResource, TestNonDefinedSimilarityFactory, PluginInfoTest, HdfsSyncSliceTest, TestSchemaSimilarityResource, TestFastLRUCache, TestIndexingPerformance, FileUtilsTest, TestDFRSimilarityFactory, FieldAnalysisRequestHandlerTest, TestRTGBase, OutputWriterTest, TestQueryTypes, NoCacheHeaderTest, TestStressLucene, TestCoreDiscovery, TestCollapseQParserPlugin, NumericFieldsTest, TestSolrDeletionPolicy2, TestFastOutputStream, TestCSVLoader, SolrTestCaseJ4Test, SpellingQueryConverterTest, TermVectorComponentDistributedTest, TestArbitraryIndexDir, CoreMergeIndexesAdminHandlerTest, TestCloudManagedSchemaAddField, TestUniqueKeyFieldResource, HighlighterTest, StatsComponentTest, ZkNodePropsTest, TestMultiCoreConfBootstrap, BasicZkTest, ShardRoutingCustomTest, TestCodecSupport, CoreAdminCreateDiscoverTest, TestManagedSchema, TestIndexSearcher, HardAutoCommitTest, DeleteShardTest, DirectUpdateHandlerTest, TestSchemaResource, TestStressReorder, BadIndexSchemaTest, TestFiltering, MultiTermTest, TestSolrQueryParserDefaultOperatorResource, TestBinaryField, TestQuerySenderNoQuery, TestHighFrequencyDictionaryFactory, TestCollationFieldDocValues, TestReload, OverseerRolesTest, TestUpdate, TestSolr4Spatial, ConvertedLegacyTest, TestRangeQuery, DisMaxRequestHandlerTest, TestFieldResource, ShardSplitTest, RAMDirectoryFactoryTest, TestStressRecovery, XsltUpdateRequestHandlerTest, SimplePostToolTest, TestRecoveryHdfs, CollectionsAPIDistributedZkTest, TestSolrCoreProperties, TestBlendedInfixSuggestions, TestXIncludeConfig, TestImplicitCoreProperties, TestCSVResponseWriter, SolrIndexConfigTest, DebugComponentTest, ChaosMonkeySafeLeaderTest, UnloadDistributedZkTest, OpenCloseCoreStressTest, OverseerTest, ShardRoutingTest, TestReplicationHandler, ZkSolrClientTest, TestZkChroot, TestDistributedSearch, TestDistributedGrouping, ZkControllerTest, TestRealTimeGet, DistributedTermsComponentTest, SolrCoreTest, SpellCheckComponentTest, TestGroupingSearch, TestLazyCores, TestBadConfig, CoreAdminHandlerTest, TestFoldingMultitermQuery, SuggesterTSTTest, TestTrie, PolyFieldTest, SolrCoreCheckLockOnStartupTest, FieldMutatingUpdateProcessorTest, SortByFunctionTest, DistanceFunctionTest, IndexBasedSpellCheckerTest, StandardRequestHandlerTest, DocumentAnalysisRequestHandlerTest, FileBasedSpellCheckerTest, XmlUpdateRequestHandlerTest, TestValueSourceCache, MoreLikeThisHandlerTest, TestSolrQueryParser, IndexSchemaRuntimeFieldTest, SolrPluginUtilsTest, TestJmxIntegration, UpdateRequestProcessorFactoryTest, MBeansHandlerTest, JsonLoaderTest, TestPHPSerializedResponseWriter, TestComponentsName, TestBinaryResponseWriter, SOLR749Test, AlternateDirectoryTest, ResponseLogComponentTest, SampleTest, MinimalSchemaTest, TestFuzzyAnalyzedSuggestions, NotRequiredUniqueKeyTest, TestPhraseSuggestions, SpellPossibilityIteratorTest, TestNumberUtils, TestIBSimilarityFactory, TestDefaultSimilarityFactory, OpenExchangeRatesOrgProviderTest, ChaosMonkeyNothingIsSafeTest]
   [junit4] Completed on J1 in 273.12s, 1 test, 1 failure <<< FAILURES!

[...truncated 19 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:453: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:433: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1268: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:901: There were test failures: 372 suites, 1604 tests, 1 failure, 35 ignored (7 assumptions)

Total time: 46 minutes 57 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0-fcs-b129 -XX:-UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure