You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2014/11/06 20:04:12 UTC

[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 677 - Still Failing

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/677/

5 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([FEA2EACA01B8699E]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([FEA2EACA01B8699E]:0)


REGRESSION:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
There are still nodes recoverying - waited for 330 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 330 seconds
	at __randomizedtesting.SeedInfo.seed([FEA2EACA01B8699E:7F4464D276E709A2]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:178)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:137)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:132)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:834)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.createCollection(CollectionsAPIDistributedZkTest.java:1332)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.addReplicaTest(CollectionsAPIDistributedZkTest.java:1259)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:210)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


REGRESSION:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testDistribSearch

Error Message:
Error executing query

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Error executing query
	at __randomizedtesting.SeedInfo.seed([FEA2EACA01B8699E:7F4464D276E709A2]:0)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:100)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.cloud.CloudInspectUtil.compareResults(CloudInspectUtil.java:223)
	at org.apache.solr.cloud.CloudInspectUtil.compareResults(CloudInspectUtil.java:165)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingBatchPerRequestWithHttpSolrServer(FullSolrCloudDistribCmdsTest.java:414)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.doTest(FullSolrCloudDistribCmdsTest.java:144)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.http.ParseException: Invalid content type: 
	at org.apache.http.entity.ContentType.parse(ContentType.java:233)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:513)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
	... 46 more


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload

Error Message:
expected:<[{indexVersion=1415292543590,generation=2,filelist=[_14q.fdt, _14q.fdx, _14q.fnm, _14q.nvd, _14q.nvm, _14q.si, _14q_Asserting_0.doc, _14q_Asserting_0.tim, _14q_Asserting_0.tip, _14r.cfe, _14r.cfs, _14r.si, _14s.cfe, _14s.cfs, _14s.si, _14t.cfe, _14t.cfs, _14t.si, _14u.cfe, _14u.cfs, _14u.si, _14v.cfe, _14v.cfs, _14v.si, _14w.cfe, _14w.cfs, _14w.si, _14x.cfe, _14x.cfs, _14x.si, _14y.cfe, _14y.cfs, _14y.si, _14z.cfe, _14z.cfs, _14z.si, _150.cfe, _150.cfs, _150.si, _151.cfe, _151.cfs, _151.si, _152.cfe, _152.cfs, _152.si, _153.cfe, _153.cfs, _153.si, _154.cfe, _154.cfs, _154.si, _155.cfe, _155.cfs, _155.si, _156.cfe, _156.cfs, _156.si, _157.cfe, _157.cfs, _157.si, _158.cfe, _158.cfs, _158.si, _159.cfe, _159.cfs, _159.si, _15a.cfe, _15a.cfs, _15a.si, _15b.cfe, _15b.cfs, _15b.si, _15c.cfe, _15c.cfs, _15c.si, _15d.cfe, _15d.cfs, _15d.si, _g0.fdt, _g0.fdx, _g0.fnm, _g0.nvd, _g0.nvm, _g0.si, _g0_Asserting_0.doc, _g0_Asserting_0.tim, _g0_Asserting_0.tip, _vd.fdt, _vd.fdx, _vd.fnm, _vd.nvd, _vd.nvm, _vd.si, _vd_Asserting_0.doc, _vd_Asserting_0.tim, _vd_Asserting_0.tip, segments_2]}]> but was:<[{indexVersion=1415292543590,generation=3,filelist=[_15e.fdt, _15e.fdx, _15e.fnm, _15e.nvd, _15e.nvm, _15e.si, _15e_Asserting_0.doc, _15e_Asserting_0.tim, _15e_Asserting_0.tip, _g0.fdt, _g0.fdx, _g0.fnm, _g0.nvd, _g0.nvm, _g0.si, _g0_Asserting_0.doc, _g0_Asserting_0.tim, _g0_Asserting_0.tip, _vd.fdt, _vd.fdx, _vd.fnm, _vd.nvd, _vd.nvm, _vd.si, _vd_Asserting_0.doc, _vd_Asserting_0.tim, _vd_Asserting_0.tip, segments_3]}, {indexVersion=1415292543590,generation=2,filelist=[_14q.fdt, _14q.fdx, _14q.fnm, _14q.nvd, _14q.nvm, _14q.si, _14q_Asserting_0.doc, _14q_Asserting_0.tim, _14q_Asserting_0.tip, _14r.cfe, _14r.cfs, _14r.si, _14s.cfe, _14s.cfs, _14s.si, _14t.cfe, _14t.cfs, _14t.si, _14u.cfe, _14u.cfs, _14u.si, _14v.cfe, _14v.cfs, _14v.si, _14w.cfe, _14w.cfs, _14w.si, _14x.cfe, _14x.cfs, _14x.si, _14y.cfe, _14y.cfs, _14y.si, _14z.cfe, _14z.cfs, _14z.si, _150.cfe, _150.cfs, _150.si, _151.cfe, _151.cfs, _151.si, _152.cfe, _152.cfs, _152.si, _153.cfe, _153.cfs, _153.si, _154.cfe, _154.cfs, _154.si, _155.cfe, _155.cfs, _155.si, _156.cfe, _156.cfs, _156.si, _157.cfe, _157.cfs, _157.si, _158.cfe, _158.cfs, _158.si, _159.cfe, _159.cfs, _159.si, _15a.cfe, _15a.cfs, _15a.si, _15b.cfe, _15b.cfs, _15b.si, _15c.cfe, _15c.cfs, _15c.si, _15d.cfe, _15d.cfs, _15d.si, _g0.fdt, _g0.fdx, _g0.fnm, _g0.nvd, _g0.nvm, _g0.si, _g0_Asserting_0.doc, _g0_Asserting_0.tim, _g0_Asserting_0.tip, _vd.fdt, _vd.fdx, _vd.fnm, _vd.nvd, _vd.nvm, _vd.si, _vd_Asserting_0.doc, _vd_Asserting_0.tim, _vd_Asserting_0.tip, segments_2]}]>

Stack Trace:
java.lang.AssertionError: expected:<[{indexVersion=1415292543590,generation=2,filelist=[_14q.fdt, _14q.fdx, _14q.fnm, _14q.nvd, _14q.nvm, _14q.si, _14q_Asserting_0.doc, _14q_Asserting_0.tim, _14q_Asserting_0.tip, _14r.cfe, _14r.cfs, _14r.si, _14s.cfe, _14s.cfs, _14s.si, _14t.cfe, _14t.cfs, _14t.si, _14u.cfe, _14u.cfs, _14u.si, _14v.cfe, _14v.cfs, _14v.si, _14w.cfe, _14w.cfs, _14w.si, _14x.cfe, _14x.cfs, _14x.si, _14y.cfe, _14y.cfs, _14y.si, _14z.cfe, _14z.cfs, _14z.si, _150.cfe, _150.cfs, _150.si, _151.cfe, _151.cfs, _151.si, _152.cfe, _152.cfs, _152.si, _153.cfe, _153.cfs, _153.si, _154.cfe, _154.cfs, _154.si, _155.cfe, _155.cfs, _155.si, _156.cfe, _156.cfs, _156.si, _157.cfe, _157.cfs, _157.si, _158.cfe, _158.cfs, _158.si, _159.cfe, _159.cfs, _159.si, _15a.cfe, _15a.cfs, _15a.si, _15b.cfe, _15b.cfs, _15b.si, _15c.cfe, _15c.cfs, _15c.si, _15d.cfe, _15d.cfs, _15d.si, _g0.fdt, _g0.fdx, _g0.fnm, _g0.nvd, _g0.nvm, _g0.si, _g0_Asserting_0.doc, _g0_Asserting_0.tim, _g0_Asserting_0.tip, _vd.fdt, _vd.fdx, _vd.fnm, _vd.nvd, _vd.nvm, _vd.si, _vd_Asserting_0.doc, _vd_Asserting_0.tim, _vd_Asserting_0.tip, segments_2]}]> but was:<[{indexVersion=1415292543590,generation=3,filelist=[_15e.fdt, _15e.fdx, _15e.fnm, _15e.nvd, _15e.nvm, _15e.si, _15e_Asserting_0.doc, _15e_Asserting_0.tim, _15e_Asserting_0.tip, _g0.fdt, _g0.fdx, _g0.fnm, _g0.nvd, _g0.nvm, _g0.si, _g0_Asserting_0.doc, _g0_Asserting_0.tim, _g0_Asserting_0.tip, _vd.fdt, _vd.fdx, _vd.fnm, _vd.nvd, _vd.nvm, _vd.si, _vd_Asserting_0.doc, _vd_Asserting_0.tim, _vd_Asserting_0.tip, segments_3]}, {indexVersion=1415292543590,generation=2,filelist=[_14q.fdt, _14q.fdx, _14q.fnm, _14q.nvd, _14q.nvm, _14q.si, _14q_Asserting_0.doc, _14q_Asserting_0.tim, _14q_Asserting_0.tip, _14r.cfe, _14r.cfs, _14r.si, _14s.cfe, _14s.cfs, _14s.si, _14t.cfe, _14t.cfs, _14t.si, _14u.cfe, _14u.cfs, _14u.si, _14v.cfe, _14v.cfs, _14v.si, _14w.cfe, _14w.cfs, _14w.si, _14x.cfe, _14x.cfs, _14x.si, _14y.cfe, _14y.cfs, _14y.si, _14z.cfe, _14z.cfs, _14z.si, _150.cfe, _150.cfs, _150.si, _151.cfe, _151.cfs, _151.si, _152.cfe, _152.cfs, _152.si, _153.cfe, _153.cfs, _153.si, _154.cfe, _154.cfs, _154.si, _155.cfe, _155.cfs, _155.si, _156.cfe, _156.cfs, _156.si, _157.cfe, _157.cfs, _157.si, _158.cfe, _158.cfs, _158.si, _159.cfe, _159.cfs, _159.si, _15a.cfe, _15a.cfs, _15a.si, _15b.cfe, _15b.cfs, _15b.si, _15c.cfe, _15c.cfs, _15c.si, _15d.cfe, _15d.cfs, _15d.si, _g0.fdt, _g0.fdx, _g0.fnm, _g0.nvd, _g0.nvm, _g0.si, _g0_Asserting_0.doc, _g0_Asserting_0.tim, _g0_Asserting_0.tip, _vd.fdt, _vd.fdx, _vd.fnm, _vd.nvd, _vd.nvm, _vd.si, _vd_Asserting_0.doc, _vd_Asserting_0.tim, _vd_Asserting_0.tip, segments_2]}]>
	at __randomizedtesting.SeedInfo.seed([FEA2EACA01B8699E:DB75F1FA71F0679D]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:147)
	at org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload(TestReplicationHandler.java:1175)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11497 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/init-core-data-001
   [junit4]   2> 114306 T303 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 114306 T303 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 114314 T303 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 114348 T303 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 114359 T304 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 114660 T303 oasc.ZkTestServer.run start zk server on port:22662
   [junit4]   2> 114673 T303 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 114979 T303 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 115161 T310 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4942c49a name:ZooKeeperConnection Watcher:127.0.0.1:22662 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 115161 T303 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 115162 T303 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 115174 T303 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 115301 T303 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 115302 T303 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 115304 T312 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b861b68 name:ZooKeeperConnection Watcher:127.0.0.1:22662/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 115305 T303 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 115305 T303 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 115313 T303 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 115318 T303 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 115322 T303 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 115325 T303 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 115330 T303 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 115331 T303 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 115337 T303 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 115337 T303 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 115342 T303 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 115343 T303 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 115348 T303 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 115348 T303 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 115352 T303 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 115353 T303 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 115357 T303 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 115358 T303 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 115362 T303 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 115363 T303 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 115367 T303 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 115368 T303 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 115372 T303 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 115373 T303 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 115377 T303 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 115377 T303 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 115381 T303 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 115382 T303 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 115406 T303 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 115407 T303 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 115409 T314 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5ae568e3 name:ZooKeeperConnection Watcher:127.0.0.1:22662/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 115410 T303 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 115410 T303 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> Using old style solr.xml
   [junit4]   2> 115414 T303 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 115415 T303 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 115416 T316 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@624e4cd1 name:ZooKeeperConnection Watcher:127.0.0.1:22662 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 115417 T303 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 115417 T303 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 115417 T303 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 115436 T303 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 115437 T303 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 115439 T318 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d34f657 name:ZooKeeperConnection Watcher:127.0.0.1:22662/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 115439 T303 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 115440 T303 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 115442 T303 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf2/solrconfig.xml
   [junit4]   2> 115442 T303 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.xml
   [junit4]   2> 115448 T303 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
   [junit4]   2> 115449 T303 oascc.SolrZkClient.makePath makePath: /configs/conf2/schema.xml
   [junit4]   2> 115453 T303 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 115454 T303 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 115458 T303 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
   [junit4]   2> 115458 T303 oascc.SolrZkClient.makePath makePath: /configs/conf2/stopwords.txt
   [junit4]   2> 115462 T303 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
   [junit4]   2> 115463 T303 oascc.SolrZkClient.makePath makePath: /configs/conf2/protwords.txt
   [junit4]   2> 115467 T303 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
   [junit4]   2> 115467 T303 oascc.SolrZkClient.makePath makePath: /configs/conf2/currency.xml
   [junit4]   2> 115471 T303 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
   [junit4]   2> 115472 T303 oascc.SolrZkClient.makePath makePath: /configs/conf2/open-exchange-rates.json
   [junit4]   2> 115476 T303 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 115477 T303 oascc.SolrZkClient.makePath makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 115480 T303 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
   [junit4]   2> 115481 T303 oascc.SolrZkClient.makePath makePath: /configs/conf2/old_synonyms.txt
   [junit4]   2> 115485 T303 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
   [junit4]   2> 115486 T303 oascc.SolrZkClient.makePath makePath: /configs/conf2/synonyms.txt
   [junit4]   2> 115490 T303 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
   [junit4]   2> 115491 T303 oascc.SolrZkClient.makePath makePath: /configs/conf2/elevate.xml
   [junit4]   2> 116925 T303 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 116993 T303 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 116999 T303 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:22668
   [junit4]   2> 117003 T303 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 117004 T303 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 117004 T303 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-002
   [junit4]   2> 117005 T303 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-002/'
   [junit4]   2> 117040 T303 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-002/solr.xml
   [junit4]   2> 117094 T303 oasc.CoreContainer.<init> New CoreContainer 79307965
   [junit4]   2> 117094 T303 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-002/]
   [junit4]   2> 117095 T303 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 117096 T303 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 117096 T303 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 117096 T303 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 117097 T303 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 117097 T303 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 117097 T303 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 117097 T303 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 117098 T303 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 117098 T303 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 117101 T303 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 117103 T303 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 117104 T303 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 117104 T303 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 117104 T303 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:22662/solr
   [junit4]   2> 117137 T303 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 117138 T303 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 117138 T303 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 117140 T329 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2075ade1 name:ZooKeeperConnection Watcher:127.0.0.1:22662 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 117141 T303 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 117141 T303 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 117154 T303 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 117156 T331 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@322ba6d7 name:ZooKeeperConnection Watcher:127.0.0.1:22662/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 117156 T303 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 117176 T303 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 117191 T303 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 117201 T303 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 117206 T303 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 117211 T303 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 117240 T303 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 117242 T303 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:22668_
   [junit4]   2> 117251 T303 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:22668_
   [junit4]   2> 117255 T303 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 117258 T303 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 117261 T303 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 117285 T303 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:22668_
   [junit4]   2> 117286 T303 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 117289 T303 oasc.Overseer.start Overseer (id=92752555564007430-127.0.0.1:22668_-n_0000000000) starting
   [junit4]   2> 117308 T303 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 117345 T303 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 117367 T333 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 117368 T303 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 117371 T303 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 117373 T303 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 117404 T332 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 117414 T303 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 117415 T303 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 117461 T303 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 117462 T303 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 117464 T337 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6ec94dc8 name:ZooKeeperConnection Watcher:127.0.0.1:22662/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 117465 T303 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 117465 T303 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 117467 T303 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 117474 T303 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 117475 T303 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 117475 T303 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 117476 T303 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 117477 T339 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@20afaa0a name:ZooKeeperConnection Watcher:127.0.0.1:22662/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 117477 T303 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 117478 T303 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 117492 T340 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 117509 T332 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: collection1
   [junit4]   2> 117510 T332 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 117520 T332 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 2
   [junit4]   2> 117530 T332 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/collection1/state.json
   [junit4]   2> 117533 T340 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 117533 T341 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 118592 T303 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 118593 T303 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 118599 T303 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 118602 T303 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:22673
   [junit4]   2> 118605 T303 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 118606 T303 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 118606 T303 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-003
   [junit4]   2> 118606 T303 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-003/'
   [junit4]   2> 118642 T303 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-003/solr.xml
   [junit4]   2> 118697 T303 oasc.CoreContainer.<init> New CoreContainer 1226200913
   [junit4]   2> 118698 T303 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-003/]
   [junit4]   2> 118699 T303 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 118699 T303 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 118699 T303 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 118699 T303 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 118700 T303 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 118700 T303 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 118700 T303 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 118701 T303 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 118701 T303 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 118701 T303 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 118704 T303 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 118706 T303 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 118707 T303 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 118707 T303 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 118707 T303 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:22662/solr
   [junit4]   2> 118708 T303 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 118708 T303 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 118709 T303 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 118711 T352 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@77347fda name:ZooKeeperConnection Watcher:127.0.0.1:22662 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 118711 T303 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 118712 T303 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 118714 T303 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 118716 T354 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@41e8505a name:ZooKeeperConnection Watcher:127.0.0.1:22662/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 118716 T303 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 118727 T303 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 119737 T303 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:22673_
   [junit4]   2> 119739 T303 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:22673_
   [junit4]   2> 119746 T303 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 119762 T303 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 119762 T303 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 121080 T303 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 121086 T303 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 121093 T303 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 121095 T303 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:22676
   [junit4]   2> 121098 T303 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 121098 T303 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 121099 T303 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-004
   [junit4]   2> 121099 T303 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-004/'
   [junit4]   2> 121130 T303 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-004/solr.xml
   [junit4]   2> 121218 T303 oasc.CoreContainer.<init> New CoreContainer 1777797839
   [junit4]   2> 121219 T303 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-004/]
   [junit4]   2> 121220 T303 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 121220 T303 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 121221 T303 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 121221 T303 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 121221 T303 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 121221 T303 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 121222 T303 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 121222 T303 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 121222 T303 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 121223 T303 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 121225 T303 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 121228 T303 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 121228 T303 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 121228 T303 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 121229 T303 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:22662/solr
   [junit4]   2> 121229 T303 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 121229 T303 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 121230 T303 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 121233 T367 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2e40dc59 name:ZooKeeperConnection Watcher:127.0.0.1:22662 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 121233 T303 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 121234 T303 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 121237 T303 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 121239 T369 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@54505bcd name:ZooKeeperConnection Watcher:127.0.0.1:22662/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 121239 T303 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 121251 T303 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 122261 T303 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:22676_
   [junit4]   2> 122263 T303 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:22676_
   [junit4]   2> 122269 T303 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 122276 T303 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 122276 T303 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 123924 T303 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 123925 T303 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 123934 T303 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 123938 T303 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:22679
   [junit4]   2> 123942 T303 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 123943 T303 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 123943 T303 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-005
   [junit4]   2> 123943 T303 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-005/'
   [junit4]   2> 123990 T303 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-005/solr.xml
   [junit4]   2> 124074 T303 oasc.CoreContainer.<init> New CoreContainer 770659873
   [junit4]   2> 124074 T303 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-005/]
   [junit4]   2> 124076 T303 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 124076 T303 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 124077 T303 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 124077 T303 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 124077 T303 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 124078 T303 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 124078 T303 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 124079 T303 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 124079 T303 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 124080 T303 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 124083 T303 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 124087 T303 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 124087 T303 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 124088 T303 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 124088 T303 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:22662/solr
   [junit4]   2> 124088 T303 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 124089 T303 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 124090 T303 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 124092 T382 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@42e550fd name:ZooKeeperConnection Watcher:127.0.0.1:22662 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 124093 T303 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 124094 T303 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 124097 T303 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 124098 T384 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c6a5b72 name:ZooKeeperConnection Watcher:127.0.0.1:22662/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 124099 T303 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 124110 T303 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 125120 T303 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:22679_
   [junit4]   2> 125122 T303 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:22679_
   [junit4]   2> 125128 T303 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 125136 T303 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 125136 T303 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 126794 T303 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 126796 T303 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 126803 T303 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 126807 T303 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:22682
   [junit4]   2> 126811 T303 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 126811 T303 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 126812 T303 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-006
   [junit4]   2> 126812 T303 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-006/'
   [junit4]   2> 126864 T303 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-006/solr.xml
   [junit4]   2> 126947 T303 oasc.CoreContainer.<init> New CoreContainer 1469336945
   [junit4]   2> 126948 T303 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-006/]
   [junit4]   2> 126949 T303 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 126949 T303 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 126950 T303 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 126950 T303 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 126951 T303 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 126951 T303 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 126951 T303 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 126952 T303 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 126952 T303 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 126953 T303 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 126957 T303 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 126960 T303 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 126961 T303 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 126961 T303 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 126962 T303 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:22662/solr
   [junit4]   2> 126962 T303 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 126962 T303 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 126964 T303 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 126966 T397 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f4e7db name:ZooKeeperConnection Watcher:127.0.0.1:22662 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 126967 T303 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 126967 T303 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 126970 T303 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 126972 T399 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@79025994 name:ZooKeeperConnection Watcher:127.0.0.1:22662/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 126972 T303 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 126982 T303 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 127991 T303 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:22682_
   [junit4]   2> 127993 T303 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:22682_
   [junit4]   2> 128000 T303 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 128007 T303 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 128008 T303 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 128323 T349 oasha.CollectionsHandler.handleCreateAction Creating Collection : name=solrj_collection&wt=javabin&collection.configName=conf1&replicationFactor=2&action=CREATE&numShards=2&version=2&router.field=myOwnField
   [junit4]   2> 128326 T340 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/collection-queue-work state SyncConnected
   [junit4]   2> 128328 T333 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField"}
   [junit4]   2> 128335 T403 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : create , {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField"}
   [junit4]   2> 128343 T403 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/solrj_collection 
   [junit4]   2> 128344 T403 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection
   [junit4]   2> 128351 T340 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 128358 T332 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: solrj_collection
   [junit4]   2> 128358 T332 oasc.Overseer$ClusterStateUpdater.createCollection Create collection solrj_collection with shards [shard1, shard2]
   [junit4]   2> 128358 T332 oasc.Overseer$ClusterStateUpdater.createCollection state version solrj_collection 1
   [junit4]   2> 128364 T370 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 128364 T340 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 128364 T355 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 128364 T400 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 128364 T341 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 128364 T385 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 128451 T403 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection solrj_collection, shardNames [shard1, shard2] , replicationFactor : 2
   [junit4]   2> 128452 T403 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica1 as part of slice shard1 of collection solrj_collection on 127.0.0.1:22682_
   [junit4]   2> 128454 T403 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica2 as part of slice shard1 of collection solrj_collection on 127.0.0.1:22668_
   [junit4]   2> 128457 T403 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica1 as part of slice shard2 of collection solrj_collection on 127.0.0.1:22679_
   [junit4]   2> 128460 T403 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica2 as part of slice shard2 of collection solrj_collection on 127.0.0.1:22676_
   [junit4]   2> 128602 T323 oasha.CoreAdminHandler.handleCreateAction core create command name=solrj_collection_shard1_replica2&wt=javabin&shard=shard1&collection.configName=conf1&action=CREATE&numShards=2&version=2&qt=/admin/cores&collection=solrj_collection
   [junit4]   2> 128606 T374 oasha.CoreAdminHandler.handleCreateAction core create command name=solrj_collection_shard2_replica1&wt=javabin&shard=shard2&collection.configName=conf1&action=CREATE&numShards=2&version=2&qt=/admin/cores&collection=solrj_collection
   [junit4]   2> 128610 T389 oasha.CoreAdminHandler.handleCreateAction core create command name=solrj_collection_shard1_replica1&wt=javabin&shard=shard1&collection.configName=conf1&action=CREATE&numShards=2&version=2&qt=/admin/cores&collection=solrj_collection
   [junit4]   2> 128612 T361 oasha.CoreAdminHandler.handleCreateAction core create command name=solrj_collection_shard2_replica2&wt=javabin&shard=shard2&collection.configName=conf1&action=CREATE&numShards=2&version=2&qt=/admin/cores&collection=solrj_collection
   [junit4]   2> 128615 T374 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica1 state=down collection=solrj_collection
   [junit4]   2> 128616 T323 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica2 state=down collection=solrj_collection
   [junit4]   2> 128616 T389 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica1 state=down collection=solrj_collection
   [junit4]   2> 128617 T361 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica2 state=down collection=solrj_collection
   [junit4]   2> 128625 T340 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 128625 T323 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 128626 T361 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 128625 T389 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 128626 T374 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 128638 T332 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:22682",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "node_name":"127.0.0.1:22682_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "state":"down"}
   [junit4]   2> 128651 T332 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:22668",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "node_name":"127.0.0.1:22668_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "state":"down"}
   [junit4]   2> 128656 T340 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 128658 T332 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:22679",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "node_name":"127.0.0.1:22679_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "state":"down"}
   [junit4]   2> 128665 T332 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:22676",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "node_name":"127.0.0.1:22676_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "state":"down"}
   [junit4]   2> 128669 T340 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 128671 T385 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 128672 T400 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 128672 T370 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 128671 T340 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 128671 T355 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 128671 T341 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 129627 T361 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica2
   [junit4]   2> 129627 T323 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica2
   [junit4]   2> 129627 T389 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica1
   [junit4]   2> 129627 T374 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica1
   [junit4]   2> 129627 T361 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 129629 T374 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 129628 T389 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 129628 T323 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 129630 T374 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 129631 T323 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 129630 T361 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 129631 T323 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 129631 T374 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 129630 T389 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 129632 T361 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 129633 T323 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 129633 T389 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 129634 T374 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 129634 T361 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 129635 T389 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 129640 T374 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-005/solrj_collection_shard2_replica1/'
   [junit4]   2> 129640 T389 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-006/solrj_collection_shard1_replica1/'
   [junit4]   2> 129640 T323 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-002/solrj_collection_shard1_replica2/'
   [junit4]   2> 129640 T361 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-004/solrj_collection_shard2_replica2/'
   [junit4]   2> 129852 T374 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 129853 T323 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 129854 T361 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 129858 T389 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 129915 T361 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 129917 T374 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 129917 T361 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 129918 T323 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 129919 T374 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 129920 T323 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 129921 T389 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 129922 T389 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 129931 T361 oass.IndexSchema.readSchema [solrj_collection_shard2_replica2] Schema name=test
   [junit4]   2> 129948 T374 oass.IndexSchema.readSchema [solrj_collection_shard2_replica1] Schema name=test
   [junit4]   2> 129950 T323 oass.IndexSchema.readSchema [solrj_collection_shard1_replica2] Schema name=test
   [junit4]   2> 129954 T389 oass.IndexSchema.readSchema [solrj_collection_shard1_replica1] Schema name=test
   [junit4]   2> 130439 T374 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 130443 T323 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 130443 T389 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 130444 T361 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 130453 T374 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 130453 T361 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 130454 T389 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 130455 T323 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 130456 T374 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 130457 T361 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 130457 T389 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 130458 T323 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 130481 T389 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 130482 T323 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 130482 T374 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 130483 T361 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 130488 T323 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 130488 T374 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 130489 T389 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 130490 T361 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 130494 T323 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 130495 T374 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 130495 T323 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 130496 T361 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 130496 T389 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 130497 T374 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 130498 T361 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 130499 T389 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 130497 T323 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 130499 T389 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 130499 T361 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 130498 T374 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 130501 T361 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 130500 T389 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 130500 T323 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 130501 T374 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 130502 T361 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 130503 T361 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 130503 T389 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 130504 T361 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica2' using configuration from collection solrj_collection
   [junit4]   2> 130504 T374 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 130503 T323 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 130505 T374 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 130504 T361 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 130504 T389 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 130506 T361 oasc.SolrCore.<init> [solrj_collection_shard2_replica2] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-004/solrj_collection_shard2_replica2/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-004/solrj_collection_shard2_replica2/data/
   [junit4]   2> 130506 T374 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica1' using configuration from collection solrj_collection
   [junit4]   2> 130505 T323 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 130508 T374 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 130508 T361 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ac9a702
   [junit4]   2> 130507 T389 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica1' using configuration from collection solrj_collection
   [junit4]   2> 130509 T374 oasc.SolrCore.<init> [solrj_collection_shard2_replica1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-005/solrj_collection_shard2_replica1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-005/solrj_collection_shard2_replica1/data/
   [junit4]   2> 130508 T323 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica2' using configuration from collection solrj_collection
   [junit4]   2> 130510 T374 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ac9a702
   [junit4]   2> 130510 T389 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 130511 T323 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 130512 T389 oasc.SolrCore.<init> [solrj_collection_shard1_replica1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-006/solrj_collection_shard1_replica1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-006/solrj_collection_shard1_replica1/data/
   [junit4]   2> 130512 T323 oasc.SolrCore.<init> [solrj_collection_shard1_replica2] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-002/solrj_collection_shard1_replica2/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-002/solrj_collection_shard1_replica2/data/
   [junit4]   2> 130513 T389 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ac9a702
   [junit4]   2> 130513 T323 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server:

[...truncated too long message...]

junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	  65) Thread[id=2186, name=searcherExecutor-1123-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]   2> 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	  66) Thread[id=2172, name=qtp1257495698-2172 Acceptor0 SocketConnector@127.0.0.1:31806, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at java.net.PlainSocketImpl.socketAccept(Native Method)
   [junit4]   2> 	        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
   [junit4]   2> 	        at java.net.ServerSocket.implAccept(ServerSocket.java:530)
   [junit4]   2> 	        at java.net.ServerSocket.accept(ServerSocket.java:498)
   [junit4]   2> 	        at org.eclipse.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
   [junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	  67) Thread[id=1873, name=coreZkRegister-1018-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]   2> 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	  68) Thread[id=1934, name=qtp1892819812-1934 Acceptor0 SocketConnector@127.0.0.1:51454, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at java.net.PlainSocketImpl.socketAccept(Native Method)
   [junit4]   2> 	        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
   [junit4]   2> 	        at java.net.ServerSocket.implAccept(ServerSocket.java:530)
   [junit4]   2> 	        at java.net.ServerSocket.accept(ServerSocket.java:498)
   [junit4]   2> 	        at org.eclipse.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
   [junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	  69) Thread[id=1875, name=qtp1669094124-1875 Acceptor0 SocketConnector@127.0.0.1:27878, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at java.net.PlainSocketImpl.socketAccept(Native Method)
   [junit4]   2> 	        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
   [junit4]   2> 	        at java.net.ServerSocket.implAccept(ServerSocket.java:530)
   [junit4]   2> 	        at java.net.ServerSocket.accept(ServerSocket.java:498)
   [junit4]   2> 	        at org.eclipse.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
   [junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	  70) Thread[id=1913, name=qtp1383421744-1913 Acceptor0 SocketConnector@127.0.0.1:30451, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at java.net.PlainSocketImpl.socketAccept(Native Method)
   [junit4]   2> 	        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
   [junit4]   2> 	        at java.net.ServerSocket.implAccept(ServerSocket.java:530)
   [junit4]   2> 	        at java.net.ServerSocket.accept(ServerSocket.java:498)
   [junit4]   2> 	        at org.eclipse.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
   [junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50), sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=lv_LV, timezone=US/Michigan
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=72,free=123046616,total=225968128
   [junit4]   2> NOTE: All tests run in this JVM: [TestExceedMaxTermLength, TestSchemaResource, TestSolr4Spatial2, PathHierarchyTokenizerFactoryTest, TestAtomicUpdateErrorCases, TestBinaryField, TestLMDirichletSimilarityFactory, DateFieldTest, RequiredFieldsTest, AutoCommitTest, DebugComponentTest, TestStandardQParsers, DeleteReplicaTest, TestDefaultSearchFieldResource, TestSweetSpotSimilarityFactory, SolrRequestParserTest, ShowFileRequestHandlerTest, SystemInfoHandlerTest, TestQuerySenderNoQuery, DirectUpdateHandlerTest, TestSchemaSimilarityResource, ZkControllerTest, FastVectorHighlighterTest, TestCodecSupport, CachingDirectoryFactoryTest, HdfsBasicDistributedZkTest, SolrCoreCheckLockOnStartupTest, TestCollationField, TestSolrConfigHandler, SuggestComponentTest, TestDistribDocBasedVersion, TriLevelCompositeIdRoutingTest, InfoHandlerTest, TestRecovery, TestManagedStopFilterFactory, TestCollectionAPI, TestFieldTypeCollectionResource, TestDocumentBuilder, ClusterStateTest, CurrencyFieldOpenExchangeTest, BlockCacheTest, TestFreeTextSuggestions, ConnectionManagerTest, SolrCoreTest, NoCacheHeaderTest, PreAnalyzedFieldTest, TestInitParams, DocValuesMissingTest, TestLFUCache, ChaosMonkeySafeLeaderTest]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.seed=FEA2EACA01B8699E -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=lv_LV -Dtests.timezone=US/Michigan -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J1 | ChaosMonkeySafeLeaderTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([FEA2EACA01B8699E]:0)
   [junit4] Completed on J1 in 7224.57s, 1 test, 2 errors <<< FAILURES!

[...truncated 25 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:532: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:473: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:502: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1358: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:965: There were test failures: 447 suites, 1820 tests, 1 suite-level error, 2 errors, 2 failures, 75 ignored (9 assumptions)

Total time: 273 minutes 8 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #675
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 841793481 bytes
Compression is 0.0%
Took 2 min 33 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 678 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/678/

1 tests failed.
REGRESSION:  org.apache.solr.spelling.SpellCheckCollatorTest.testEstimatedHitCounts

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
	at __randomizedtesting.SeedInfo.seed([FA1D81C21BB12A9E:CBA63FF7BE8E3A4E]:0)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:723)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:690)
	at org.apache.solr.spelling.SpellCheckCollatorTest.testEstimatedHitCounts(SpellCheckCollatorTest.java:530)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//lst[@name='spellcheck']/lst[@name='collations']/lst[@name='collation']/int[@name='hits' and 6 <= . and . <= 10]
	xml response was: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">5</int></lst><result name="response" numFound="0" start="0"></result><lst name="spellcheck"><lst name="suggestions"><lst name="everother"><int name="numFound">1</int><int name="startOffset">9</int><int name="endOffset">18</int><arr name="suggestion"><str>everyother</str></arr></lst></lst><lst name="collations"><lst name="collation"><str name="collationQuery">teststop:everyother</str><int name="hits">14</int><lst name="misspellingsAndCorrections"><str name="everother">everyother</str></lst></lst></lst></lst>
</response>

	request was:spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Aeverother&spellcheck.collateMaxCollectDocs=5
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:716)
	... 42 more




Build Log:
[...truncated 11559 lines...]
   [junit4] Suite: org.apache.solr.spelling.SpellCheckCollatorTest
   [junit4]   2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.spelling.SpellCheckCollatorTest-FA1D81C21BB12A9E-001/init-core-data-001
   [junit4]   2> 881129 T1066 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 881130 T1066 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 881131 T1066 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 881132 T1066 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 881133 T1066 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 881133 T1066 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 881266 T1066 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 881315 T1066 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-spellcheckcomponent.xml
   [junit4]   2> 881316 T1066 oass.IndexSchema.readSchema Reading Solr Schema from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema.xml
   [junit4]   2> 881329 T1066 oass.IndexSchema.readSchema [null] Schema name=test
   [junit4]   2> 881722 T1066 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 881735 T1066 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 881738 T1066 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 881750 T1066 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 881755 T1066 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 881760 T1066 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 881761 T1066 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 881761 T1066 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 881762 T1066 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 881762 T1066 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 881763 T1066 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 881763 T1066 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 881763 T1066 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr
   [junit4]   2> 881764 T1066 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/'
   [junit4]   2> 881859 T1066 oasc.CoreContainer.<init> New CoreContainer 498194059
   [junit4]   2> 881860 T1066 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/]
   [junit4]   2> 881861 T1066 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 881862 T1066 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 881862 T1066 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 881862 T1066 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 881863 T1066 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 881863 T1066 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 881863 T1066 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 881864 T1066 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 881864 T1066 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 881864 T1066 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 881871 T1066 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=30000&connTimeout=30000&retry=false
   [junit4]   2> 881875 T1066 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 881876 T1066 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 881876 T1066 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 881887 T1067 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 881890 T1067 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 881891 T1067 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 881891 T1067 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 882000 T1067 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 882049 T1067 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-spellcheckcomponent.xml
   [junit4]   2> 882050 T1067 oass.IndexSchema.readSchema Reading Solr Schema from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema.xml
   [junit4]   2> 882062 T1067 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 882611 T1067 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 882623 T1067 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 882628 T1067 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 882642 T1067 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 882650 T1067 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 882658 T1067 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 882659 T1067 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 882660 T1067 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 882660 T1067 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 882661 T1067 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 882661 T1067 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 882661 T1067 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/
   [junit4]   2> 882662 T1067 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 882662 T1067 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.spelling.SpellCheckCollatorTest-FA1D81C21BB12A9E-001/init-core-data-001/
   [junit4]   2> 882662 T1067 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 882663 T1067 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.spelling.SpellCheckCollatorTest-FA1D81C21BB12A9E-001/init-core-data-001
   [junit4]   2> 882664 T1067 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.spelling.SpellCheckCollatorTest-FA1D81C21BB12A9E-001/init-core-data-001/index/
   [junit4]   2> 882664 T1067 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.spelling.SpellCheckCollatorTest-FA1D81C21BB12A9E-001/init-core-data-001/index' doesn't exist. Creating new index...
   [junit4]   2> 882665 T1067 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.spelling.SpellCheckCollatorTest-FA1D81C21BB12A9E-001/init-core-data-001/index
   [junit4]   2> 882665 T1067 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=18, maxMergeAtOnceExplicit=17, maxMergedSegmentMB=52.46484375, floorSegmentMB=1.146484375, forceMergeDeletesPctAllowed=23.516122936869646, segmentsPerTier=12.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 882667 T1067 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-5f1a1a70)),segFN=segments_1,generation=1}
   [junit4]   2> 882667 T1067 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 882669 T1067 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 882670 T1067 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 882670 T1067 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 882671 T1067 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 882671 T1067 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 882671 T1067 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 882672 T1067 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 882672 T1067 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 882673 T1067 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 882673 T1067 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 882716 T1067 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 882720 T1067 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 882724 T1067 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 882728 T1067 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 882733 T1067 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 882733 T1067 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 882734 T1067 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=24, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.212883020738306]
   [junit4]   2> 882734 T1067 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-5f1a1a70)),segFN=segments_1,generation=1}
   [junit4]   2> 882734 T1067 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 882735 T1067 oass.SolrIndexSearcher.<init> Opening Searcher@229cab45[collection1] main
   [junit4]   2> 882736 T1067 oasr.ManagedResourceStorage.newStorageIO WARN Cannot write to config directory /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf; switching to use InMemory storage instead.
   [junit4]   2> 882736 T1067 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 882736 T1067 oasr.ManagedResourceStorage.load Reading _rest_managed.json using InMemoryStorage
   [junit4]   2> 882737 T1067 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
   [junit4]   2> 882737 T1067 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using InMemoryStorage
   [junit4]   2> 882737 T1067 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 882738 T1067 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 882764 T1067 oashc.SpellCheckComponent.inform Registering newSearcher listener for spellchecker: default
   [junit4]   2> 882783 T1067 oashc.SpellCheckComponent.inform Registering newSearcher listener for spellchecker: default_teststop
   [junit4]   2> 883024 T1067 oass.DirectSolrSpellChecker.init init: {name=direct,classname=solr.DirectSolrSpellChecker,minQueryLength=3,maxQueryFrequency=100.0,field=teststop}
   [junit4]   2> 883025 T1067 oass.DirectSolrSpellChecker.init init: {name=direct_lowerfilt,classname=solr.DirectSolrSpellChecker,minQueryLength=3,maxQueryFrequency=100.0,field=lowerfilt}
   [junit4]   2> 883043 T1067 oashc.SpellCheckComponent.inform Registering newSearcher listener for spellchecker: threshold
   [junit4]   2> 883043 T1067 oass.DirectSolrSpellChecker.init init: {name=threshold_direct,classname=solr.DirectSolrSpellChecker,field=lowerfilt,spellcheckIndexDir=spellcheckerThreshold,buildOnCommit=true,thresholdTokenFrequency=0.29}
   [junit4]   2> 883043 T1067 oashc.SpellCheckComponent.inform Registering newSearcher listener for spellchecker: threshold_direct
   [junit4]   2> 883053 T1067 oashc.SpellCheckComponent.inform Registering newSearcher listener for spellchecker: multipleFields
   [junit4]   2> 883081 T1067 oashc.SpellCheckComponent.inform Registering newSearcher listener for spellchecker: freq
   [junit4]   2> 883091 T1067 oashc.SpellCheckComponent.inform Registering newSearcher listener for spellchecker: fqcn
   [junit4]   2> 883092 T1068 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@229cab45[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 883092 T1067 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 883095 T1066 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> ASYNC  NEW_CORE C329 name=collection1 org.apache.solr.core.SolrCore@363154a2
   [junit4]   2> 883098 T1066 C329 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[0]} 0 2
   [junit4]   2> 883134 T1066 C329 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[1]} 0 7
   [junit4]   2> 883138 T1066 C329 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[2]} 0 1
   [junit4]   2> 883145 T1066 C329 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[3]} 0 5
   [junit4]   2> 883148 T1066 C329 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[4]} 0 1
   [junit4]   2> 883156 T1066 C329 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[5]} 0 6
   [junit4]   2> 883159 T1066 C329 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[6]} 0 1
   [junit4]   2> 883165 T1066 C329 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[7]} 0 4
   [junit4]   2> 883169 T1066 C329 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[8]} 0 2
   [junit4]   2> 883174 T1066 C329 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[9]} 0 4
   [junit4]   2> 883177 T1066 C329 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[10]} 0 1
   [junit4]   2> 883183 T1066 C329 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[11]} 0 4
   [junit4]   2> 883186 T1066 C329 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[12]} 0 2
   [junit4]   2> 883198 T1066 C329 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[13]} 0 11
   [junit4]   2> 883202 T1066 C329 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[14]} 0 2
   [junit4]   2> 883207 T1066 C329 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[15]} 0 3
   [junit4]   2> 883211 T1066 C329 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[16]} 0 2
   [junit4]   2> 883213 T1066 C329 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 883315 T1066 C329 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-5f1a1a70)),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-5f1a1a70)),segFN=segments_2,generation=2}
   [junit4]   2> 883316 T1066 C329 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 883321 T1066 C329 oass.SolrIndexSearcher.<init> Opening Searcher@553bb40d[collection1] main
   [junit4]   2> 883322 T1066 C329 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 883322 T1068 oashc.SpellCheckComponent$SpellCheckerListener.buildSpellIndex Building spell index for spell checker: default
   [junit4]   2> 883429 T1068 oashc.SpellCheckComponent$SpellCheckerListener.buildSpellIndex Building spell index for spell checker: default_teststop
   [junit4]   2> 883505 T1068 oashc.SpellCheckComponent$SpellCheckerListener.buildSpellIndex Building spell index for spell checker: threshold
   [junit4]   2> 883518 T1068 oashc.SpellCheckComponent$SpellCheckerListener.buildSpellIndex Building spell index for spell checker: threshold_direct
   [junit4]   2> 883518 T1068 oashc.SpellCheckComponent$SpellCheckerListener.buildSpellIndex Building spell index for spell checker: multipleFields
   [junit4]   2> 883584 T1068 oashc.SpellCheckComponent$SpellCheckerListener.buildSpellIndex Building spell index for spell checker: freq
   [junit4]   2> 883648 T1068 oashc.SpellCheckComponent$SpellCheckerListener.buildSpellIndex Building spell index for spell checker: fqcn
   [junit4]   2> 883707 T1068 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@553bb40d[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):C2) Uninverting(_1(6.0.0):C2) Uninverting(_2(6.0.0):C2) Uninverting(_3(6.0.0):C2) Uninverting(_4(6.0.0):C2) Uninverting(_5(6.0.0):C2) Uninverting(_6(6.0.0):C2) Uninverting(_7(6.0.0):C2) Uninverting(_8(6.0.0):C1)))}
   [junit4]   2> 883707 T1066 C329 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 494
   [junit4]   2> 883715 T1066 oas.SolrTestCaseJ4.setUp ###Starting testCollateWithOverride
   [junit4]   2> 884058 T1066 C329 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck.count=10&qt=spellCheckCompRH&mm=1&spellcheck.maxCollationTries=10&spellcheck.collate=true&wt=xml&qf=teststop&spellcheck.dictionary=direct&q=partisian+politcal+mashine&spellcheck=true&spellcheck.maxCollations=10&defType=edismax} hits=0 status=0 QTime=342 
   [junit4]   2> 884113 T1066 C329 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck.collateParam.mm=100%25&spellcheck.count=10&qt=spellCheckCompRH&mm=1&spellcheck.maxCollationTries=10&spellcheck.collate=true&wt=xml&qf=teststop&spellcheck.dictionary=direct&q=partisian+politcal+mashine&spellcheck=true&spellcheck.maxCollations=10&defType=edismax} hits=0 status=0 QTime=48 
   [junit4]   2> 884116 T1066 oas.SolrTestCaseJ4.tearDown ###Ending testCollateWithOverride
   [junit4]   2> 884124 T1066 oas.SolrTestCaseJ4.setUp ###Starting testCollateWithGrouping
   [junit4]   2> 884243 T1066 oas.SolrTestCaseJ4.tearDown ###Ending testCollateWithGrouping
   [junit4]   2> 884251 T1066 oas.SolrTestCaseJ4.setUp ###Starting testCollationWithHypens
   [junit4]   2> 884623 T1066 oas.SolrTestCaseJ4.tearDown ###Ending testCollationWithHypens
   [junit4]   2> 884632 T1066 oas.SolrTestCaseJ4.setUp ###Starting testCollateWithMultipleRequestHandlers
   [junit4]   2> 884712 T1066 oas.SolrTestCaseJ4.tearDown ###Ending testCollateWithMultipleRequestHandlers
   [junit4]   2> 884720 T1066 oas.SolrTestCaseJ4.setUp ###Starting testContextSensitiveCollate
   [junit4]   2> 884743 T1066 C329 oasc.SolrCore.execute [collection1] webapp=null path=null params={qt=spellCheckCompRH&spellcheck.collate=true&wt=xml&indent=true&spellcheck.collateExtendedResults=true&spellcheck.build=true&q=teststop:(flew+AND+form+AND+heathrow)&spellcheck=true&spellcheck.maxResultsForSuggest=0&spellcheck.count=10&spellcheck.alternativeTermCount=5&spellcheck.extendedResults=true&spellcheck.maxCollationTries=10&spellcheck.dictionary=direct&spellcheck.maxCollations=1} hits=0 status=0 QTime=23 
   [junit4]   2> 884766 T1066 C329 oasc.SolrCore.execute [collection1] webapp=null path=null params={qt=spellCheckCompRH&spellcheck.collate=true&wt=xml&indent=true&spellcheck.collateExtendedResults=true&q=teststop:(june+AND+customs)&spellcheck=true&spellcheck.maxResultsForSuggest=1&spellcheck.count=10&spellcheck.alternativeTermCount=5&spellcheck.extendedResults=true&spellcheck.maxCollationTries=10&spellcheck.dictionary=direct&spellcheck.maxCollations=1} hits=1 status=0 QTime=14 
   [junit4]   2> 884778 T1066 C329 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck.count=10&spellcheck.alternativeTermCount=0&qt=spellCheckCompRH&mm=2&spellcheck.collate=true&wt=xml&indent=true&spellcheck.dictionary=direct&spellcheck=true&q=teststop:(june+customs)} hits=2 status=0 QTime=4 
   [junit4]   2> 884901 T1066 C329 oasc.SolrCore.execute [collection1] webapp=null path=null params={qt=spellCheckCompRH&spellcheck.collate=true&wt=xml&indent=true&spellcheck.collateExtendedResults=true&spellcheck.build=true&q=teststop:(flew+AND+form+AND+heathrow)&spellcheck=true&spellcheck.maxResultsForSuggest=0&spellcheck.count=10&spellcheck.alternativeTermCount=5&spellcheck.extendedResults=true&spellcheck.maxCollationTries=10&spellcheck.dictionary=default_teststop&spellcheck.maxCollations=1} hits=0 status=0 QTime=122 
   [junit4]   2> 884921 T1066 C329 oasc.SolrCore.execute [collection1] webapp=null path=null params={qt=spellCheckCompRH&spellcheck.collate=true&wt=xml&indent=true&spellcheck.collateExtendedResults=true&q=teststop:(june+AND+customs)&spellcheck=true&spellcheck.maxResultsForSuggest=1&spellcheck.count=10&spellcheck.alternativeTermCount=5&spellcheck.extendedResults=true&spellcheck.maxCollationTries=10&spellcheck.dictionary=default_teststop&spellcheck.maxCollations=1} hits=1 status=0 QTime=9 
   [junit4]   2> 884934 T1066 C329 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck.count=10&spellcheck.alternativeTermCount=0&qt=spellCheckCompRH&mm=2&spellcheck.collate=true&wt=xml&indent=true&spellcheck.dictionary=default_teststop&spellcheck=true&q=teststop:(june+customs)} hits=2 status=0 QTime=4 
   [junit4]   2> 884935 T1066 oas.SolrTestCaseJ4.tearDown ###Ending testContextSensitiveCollate
   [junit4]   2> 884944 T1066 oas.SolrTestCaseJ4.setUp ###Starting testEstimatedHitCounts
   [junit4]   2> 884956 T1066 C329 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia} hits=0 status=0 QTime=12 
   [junit4]   2> 884969 T1066 C329 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=0} hits=0 status=0 QTime=8 
   [junit4]   2> 884982 T1066 C329 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=30} hits=0 status=0 QTime=9 
   [junit4]   2> 884997 T1066 C329 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=100} hits=0 status=0 QTime=10 
   [junit4]   2> 885009 T1066 C329 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=10000} hits=0 status=0 QTime=9 
   [junit4]   2> 885022 T1066 C329 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=11} hits=0 status=0 QTime=9 
   [junit4]   2> 885036 T1066 C329 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=14} hits=0 status=0 QTime=9 
   [junit4]   2> 885050 T1066 C329 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=4} hits=0 status=0 QTime=10 
   [junit4]   2> 885064 T1066 C329 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=4} hits=0 status=0 QTime=10 
   [junit4]   2> 885078 T1066 C329 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=8} hits=0 status=0 QTime=10 
   [junit4]   2> 885092 T1066 C329 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=10} hits=0 status=0 QTime=11 
   [junit4]   2> 885105 T1066 C329 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=8} hits=0 status=0 QTime=9 
   [junit4]   2> 885120 T1066 C329 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=3} hits=0 status=0 QTime=10 
   [junit4]   2> 885133 T1066 C329 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=9} hits=0 status=0 QTime=9 
   [junit4]   2> 885148 T1066 C329 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=7} hits=0 status=0 QTime=10 
   [junit4]   2> 885162 T1066 C329 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=11} hits=0 status=0 QTime=10 
   [junit4]   2> 885178 T1066 C329 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=8} hits=0 status=0 QTime=11 
   [junit4]   2> 885191 T1066 C329 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=10} hits=0 status=0 QTime=9 
   [junit4]   2> 885204 T1066 C329 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=10} hits=0 status=0 QTime=9 
   [junit4]   2> 885218 T1066 C329 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=12} hits=0 status=0 QTime=10 
   [junit4]   2> 885233 T1066 C329 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=8} hits=0 status=0 QTime=11 
   [junit4]   2> 885249 T1066 C329 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=8} hits=0 status=0 QTime=11 
   [junit4]   2> 885264 T1066 C329 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=12} hits=0 status=0 QTime=10 
   [junit4]   2> 885281 T1066 C329 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=14} hits=0 status=0 QTime=12 
   [junit4]   2> ASYNC  NEW_CORE C330 name=collection1 org.apache.solr.core.SolrCore@363154a2
   [junit4]   2> 885305 T1066 C330 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=10} hits=0 status=0 QTime=19 
   [junit4]   2> 885319 T1066 C330 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=17} hits=0 status=0 QTime=10 
   [junit4]   2> 885332 T1066 C330 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=8} hits=0 status=0 QTime=9 
   [junit4]   2> 885345 T1066 C330 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=16} hits=0 status=0 QTime=9 
   [junit4]   2> 885358 T1066 C330 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=14} hits=0 status=0 QTime=9 
   [junit4]   2> 885371 T1066 C330 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=1} hits=0 status=0 QTime=9 
   [junit4]   2> 885384 T1066 C330 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=5} hits=0 status=0 QTime=9 
   [junit4]   2> 885397 T1066 C330 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=5} hits=0 status=0 QTime=9 
   [junit4]   2> 885406 T1066 C330 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=1} hits=0 status=0 QTime=6 
   [junit4]   2> 885415 T1066 C330 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=2} hits=0 status=0 QTime=6 
   [junit4]   2> 885423 T1066 C330 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=4} hits=0 status=0 QTime=6 
   [junit4]   2> 885433 T1066 C330 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=13} hits=0 status=0 QTime=7 
   [junit4]   2> 885443 T1066 C330 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=16} hits=0 status=0 QTime=7 
   [junit4]   2> 885453 T1066 C330 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=12} hits=0 status=0 QTime=7 
   [junit4]   2> 885464 T1066 C330 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=15} hits=0 status=0 QTime=8 
   [junit4]   2> 885475 T1066 C330 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=15} hits=0 status=0 QTime=7 
   [junit4]   2> 885486 T1066 C330 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=12} hits=0 status=0 QTime=8 
   [junit4]   2> 885496 T1066 C330 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=5} hits=0 status=0 QTime=8 
   [junit4]   2> 885508 T1066 C330 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=15} hits=0 status=0 QTime=8 
   [junit4]   2> 885518 T1066 C330 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=11} hits=0 status=0 QTime=7 
   [junit4]   2> 885528 T1066 C330 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=6} hits=0 status=0 QTime=7 
   [junit4]   2> 885537 T1066 C330 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=17} hits=0 status=0 QTime=6 
   [junit4]   2> 885547 T1066 C330 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=7} hits=0 status=0 QTime=7 
   [junit4]   2> 885556 T1066 C330 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=9} hits=0 status=0 QTime=6 
   [junit4]   2> 885564 T1066 C330 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=9} hits=0 status=0 QTime=6 
   [junit4]   2> 885572 T1066 C330 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=11} hits=0 status=0 QTime=5 
   [junit4]   2> 885581 T1066 C330 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=15} hits=0 status=0 QTime=5 
   [junit4]   2> 885590 T1066 C330 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=7} hits=0 status=0 QTime=6 
   [junit4]   2> 885598 T1066 C330 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=14} hits=0 status=0 QTime=5 
   [junit4]   2> 885606 T1066 C330 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=10} hits=0 status=0 QTime=5 
   [junit4]   2> 885614 T1066 C330 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=5} hits=0 status=0 QTime=5 
   [junit4]   2> 885622 T1066 C330 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Ametnoia&spellcheck.collateMaxCollectDocs=16} hits=0 status=0 QTime=5 
   [junit4]   2> 885631 T1066 C330 oasc.SolrCore.execute [collection1] webapp=null path=null params={spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Aeverother&spellcheck.collateMaxCollectDocs=5} hits=0 status=0 QTime=5 
   [junit4]   2> 885634 T1066 oas.SolrTestCaseJ4.assertQ ERROR REQUEST FAILED: xpath=//lst[@name='spellcheck']/lst[@name='collations']/lst[@name='collation']/int[@name='hits' and 6 <= . and . <= 10]
   [junit4]   2> 		xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]   2> 	<response>
   [junit4]   2> 	<lst name="responseHeader"><int name="status">0</int><int name="QTime">5</int></lst><result name="response" numFound="0" start="0"></result><lst name="spellcheck"><lst name="suggestions"><lst name="everother"><int name="numFound">1</int><int name="startOffset">9</int><int name="endOffset">18</int><arr name="suggestion"><str>everyother</str></arr></lst></lst><lst name="collations"><lst name="collation"><str name="collationQuery">teststop:everyother</str><int name="hits">14</int><lst name="misspellingsAndCorrections"><str name="everother">everyother</str></lst></lst></lst></lst>
   [junit4]   2> 	</response>
   [junit4]   2> 	
   [junit4]   2> 		request was:spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Aeverother&spellcheck.collateMaxCollectDocs=5
   [junit4]   2> 885636 T1066 oasc.SolrException.log ERROR REQUEST FAILED: spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Aeverother&spellcheck.collateMaxCollectDocs=5:java.lang.RuntimeException: REQUEST FAILED: xpath=//lst[@name='spellcheck']/lst[@name='collations']/lst[@name='collation']/int[@name='hits' and 6 <= . and . <= 10]
   [junit4]   2> 		xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]   2> 	<response>
   [junit4]   2> 	<lst name="responseHeader"><int name="status">0</int><int name="QTime">5</int></lst><result name="response" numFound="0" start="0"></result><lst name="spellcheck"><lst name="suggestions"><lst name="everother"><int name="numFound">1</int><int name="startOffset">9</int><int name="endOffset">18</int><arr name="suggestion"><str>everyother</str></arr></lst></lst><lst name="collations"><lst name="collation"><str name="collationQuery">teststop:everyother</str><int name="hits">14</int><lst name="misspellingsAndCorrections"><str name="everother">everyother</str></lst></lst></lst></lst>
   [junit4]   2> 	</response>
   [junit4]   2> 	
   [junit4]   2> 		request was:spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Aeverother&spellcheck.collateMaxCollectDocs=5
   [junit4]   2> 		at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:716)
   [junit4]   2> 		at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:690)
   [junit4]   2> 		at org.apache.solr.spelling.SpellCheckCollatorTest.testEstimatedHitCounts(SpellCheckCollatorTest.java:530)
   [junit4]   2> 		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 		at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   2> 		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 		at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   2> 		at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 		at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   2> 		at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	
   [junit4]   2> 885638 T1066 oas.SolrTestCaseJ4.tearDown ###Ending testEstimatedHitCounts
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SpellCheckCollatorTest -Dtests.method=testEstimatedHitCounts -Dtests.seed=FA1D81C21BB12A9E -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=iw -Dtests.timezone=Europe/Minsk -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.75s J2 | SpellCheckCollatorTest.testEstimatedHitCounts <<<
   [junit4]    > Throwable #1: java.lang.RuntimeException: Exception during query
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([FA1D81C21BB12A9E:CBA63FF7BE8E3A4E]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:723)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:690)
   [junit4]    > 	at org.apache.solr.spelling.SpellCheckCollatorTest.testEstimatedHitCounts(SpellCheckCollatorTest.java:530)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//lst[@name='spellcheck']/lst[@name='collations']/lst[@name='collation']/int[@name='hits' and 6 <= . and . <= 10]
   [junit4]    > 	xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]    > <response>
   [junit4]    > <lst name="responseHeader"><int name="status">0</int><int name="QTime">5</int></lst><result name="response" numFound="0" start="0"></result><lst name="spellcheck"><lst name="suggestions"><lst name="everother"><int name="numFound">1</int><int name="startOffset">9</int><int name="endOffset">18</int><arr name="suggestion"><str>everyother</str></arr></lst></lst><lst name="collations"><lst name="collation"><str name="collationQuery">teststop:everyother</str><int name="hits">14</int><lst name="misspellingsAndCorrections"><str name="everother">everyother</str></lst></lst></lst></lst>
   [junit4]    > </response>
   [junit4]    > 	request was:spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=spellCheckCompRH&q=teststop%3Aeverother&spellcheck.collateMaxCollectDocs=5
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:716)
   [junit4]    > 	... 42 more
   [junit4]   2> 885692 T1066 oas.SolrTestCaseJ4.setUp ###Starting testZeroTries
   [junit4]   2> 885784 T1066 oas.SolrTestCaseJ4.tearDown ###Ending testZeroTries
   [junit4]   2> 885792 T1066 oas.SolrTestCaseJ4.setUp ###Starting testCollateWithFilter
   [junit4]   2> 885889 T1066 oas.SolrTestCaseJ4.tearDown ###Ending testCollateWithFilter
   [junit4]   2> 885898 T1066 oas.SolrTestCaseJ4.setUp ###Starting testExtendedCollate
   [junit4]   2> 885993 T1066 oas.SolrTestCaseJ4.tearDown ###Ending testExtendedCollate
   [junit4]   2> 885995 T1066 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 885995 T1066 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=498194059
   [junit4]   2> 885995 T1066 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@363154a2
   [junit4]   2> 885996 T1066 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=17,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 885996 T1066 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 885996 T1066 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 885996 T1066 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 885998 T1066 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 885999 T1066 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 885999 T1066 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.spelling.SpellCheckCollatorTest-FA1D81C21BB12A9E-001/init-core-data-001 [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.spelling.SpellCheckCollatorTest-FA1D81C21BB12A9E-001/init-core-data-001;done=false>>]
   [junit4]   2> 885999 T1066 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.spelling.SpellCheckCollatorTest-FA1D81C21BB12A9E-001/init-core-data-001
   [junit4]   2> 885999 T1066 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.spelling.SpellCheckCollatorTest-FA1D81C21BB12A9E-001/init-core-data-001/index [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.spelling.SpellCheckCollatorTest-FA1D81C21BB12A9E-001/init-core-data-001/index;done=false>>]
   [junit4]   2> 885999 T1066 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.spelling.SpellCheckCollatorTest-FA1D81C21BB12A9E-001/init-core-data-001/index
   [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.spelling.SpellCheckCollatorTest-FA1D81C21BB12A9E-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {teststop=PostingsFormat(name=MockRandom), id=PostingsFormat(name=SimpleText), lowerfilt1and2=PostingsFormat(name=MockRandom), word=PostingsFormat(name=MockRandom), gram1=PostingsFormat(name=MockRandom), start1=PostingsFormat(name=SimpleText), gram3=PostingsFormat(name=Memory doPackFST= false), gram2=PostingsFormat(name=SimpleText), range_facet_l=PostingsFormat(name=MockRandom), end3=PostingsFormat(name=Memory doPackFST= false), lowerfilt1=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), end1=PostingsFormat(name=MockRandom), intDefault=PostingsFormat(name=Memory doPackFST= false), gram4=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), start3=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), end4=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), timestamp=PostingsFormat(name=MockRandom), end2=PostingsFormat(name=SimpleText), start4=PostingsFormat(name=Memory doPackFST= false), multiDefault=PostingsFormat(name=MockRandom), start2=PostingsFormat(name=MockRandom), lowerfilt=PostingsFormat(name=MockRandom)}, docValues:{timestamp=DocValuesFormat(name=SimpleText)}, sim=DefaultSimilarity, locale=iw, timezone=Europe/Minsk
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=102691888,total=285212672
   [junit4]   2> NOTE: All tests run in this JVM: [TestCharFilters, TestPerFieldSimilarity, TestQuerySenderListener, TestReversedWildcardFilterFactory, TestPivotHelperCode, TestRawResponseWriter, TestStressUserVersions, EnumFieldTest, TestCursorMarkWithoutUniqueKey, TestSolrCoreProperties, MultiTermTest, XmlUpdateRequestHandlerTest, TestRestManager, ZkNodePropsTest, TestCollapseQParserPlugin, DateRangeFieldTest, SimpleMLTQParserTest, TestJmxMonitoredMap, CloudMLTQParserTest, TestTolerantSearch, ShardSplitTest, HardAutoCommitTest, BasicDistributedZk2Test, SpellCheckCollatorTest]
   [junit4] Completed on J2 in 4.90s, 9 tests, 1 error <<< FAILURES!

[...truncated 234 lines...]
   [junit4] JVM J1: stdout was not empty, see: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/temp/junit4-J1-20141107_151248_360.sysout
   [junit4] >>> JVM J1: stdout (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Dumping heap to /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/heapdumps/java_pid88993.hprof ...
   [junit4] Heap dump file created [545477326 bytes in 7.242 secs]
   [junit4] <<< JVM J1: EOF ----

   [junit4] JVM J1: stderr was not empty, see: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/temp/junit4-J1-20141107_151248_360.syserr
   [junit4] >>> JVM J1: stderr (verbatim) ----
   [junit4] WARN: Unhandled exception in event serialization. -> java.lang.OutOfMemoryError: GC overhead limit exceeded (stack unavailable; OOM)
   [junit4] <<< JVM J1: EOF ----

[...truncated 766 lines...]
   [junit4] ERROR: JVM J1 ended with an exception, command line: /usr/local/openjdk7/jre/bin/java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/heapdumps -ea -esa -Dtests.prefix=tests -Dtests.seed=FA1D81C21BB12A9E -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.luceneMatchVersion=6.0.0 -Dtests.cleanthreads=perClass -Djava.util.logging.config.file=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/tools/junit4/logging.properties -Dtests.nightly=true -Dtests.weekly=false -Dtests.monster=false -Dtests.slow=true -Dtests.asserts=true -Dtests.multiplier=2 -DtempDir=./temp -Djava.io.tmpdir=./temp -Djunit4.tempDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/temp -Dclover.db.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build/clover/db -Djava.security.policy=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/tools/junit4/tests.policy -Dtests.LUCENE_VERSION=6.0.0 -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Dtests.jettyConnector=Socket -Dtests.leaveTemporary=false -Dtests.filterstacks=true -Dtests.disableHdfs=true -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -Dfile.encoding=UTF-8 -classpath /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/classes/test:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-test-framework/classes/java:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/test-framework/lib/junit4-ant-2.1.9.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build/test-framework/classes/java:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build/codecs/classes/java:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/classes/java:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/classes/java:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build/analysis/common/lucene-analyzers-common-6.0.0-SNAPSHOT.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build/analysis/kuromoji/lucene-analyzers-kuromoji-6.0.0-SNAPSHOT.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build/analysis/phonetic/lucene-analyzers-phonetic-6.0.0-SNAPSHOT.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build/codecs/lucene-codecs-6.0.0-SNAPSHOT.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build/backward-codecs/lucene-backward-codecs-6.0.0-SNAPSHOT.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build/highlighter/lucene-highlighter-6.0.0-SNAPSHOT.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build/memory/lucene-memory-6.0.0-SNAPSHOT.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build/misc/lucene-misc-6.0.0-SNAPSHOT.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build/spatial/lucene-spatial-6.0.0-SNAPSHOT.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build/expressions/lucene-expressions-6.0.0-SNAPSHOT.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build/suggest/lucene-suggest-6.0.0-SNAPSHOT.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build/grouping/lucene-grouping-6.0.0-SNAPSHOT.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build/queries/lucene-queries-6.0.0-SNAPSHOT.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build/queryparser/lucene-queryparser-6.0.0-SNAPSHOT.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build/join/lucene-join-6.0.0-SNAPSHOT.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/lib/antlr-runtime-3.5.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/lib/asm-4.1.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/lib/asm-commons-4.1.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/lib/commons-cli-1.2.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/lib/commons-codec-1.9.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/lib/commons-configuration-1.6.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/lib/commons-fileupload-1.2.1.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/lib/commons-lang-2.6.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/lib/concurrentlinkedhashmap-lru-1.2.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/lib/dom4j-1.6.1.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/lib/guava-14.0.1.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/lib/hadoop-annotations-2.2.0.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/lib/hadoop-auth-2.2.0.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/lib/hadoop-common-2.2.0.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/lib/hadoop-hdfs-2.2.0.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/lib/hppc-0.5.2.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/lib/joda-time-2.2.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/lib/log4j-1.2.17.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/lib/org.restlet-2.1.1.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/lib/org.restlet.ext.servlet-2.1.1.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/lib/protobuf-java-2.5.0.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/lib/slf4j-log4j12-1.7.6.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/lib/spatial4j-0.4.1.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/solrj/lib/commons-io-2.4.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/solrj/lib/httpclient-4.3.1.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/solrj/lib/httpcore-4.3.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/solrj/lib/httpmime-4.3.1.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/solrj/lib/noggit-0.6.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/solrj/lib/slf4j-api-1.7.6.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/solrj/lib/wstx-asl-3.2.7.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/solrj/lib/zookeeper-3.4.6.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/server/lib/jetty-continuation-8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/server/lib/jetty-deploy-8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/server/lib/jetty-http-8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/server/lib/jetty-io-8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/server/lib/jetty-jmx-8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/server/lib/jetty-security-8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/server/lib/jetty-server-8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/server/lib/jetty-servlet-8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/server/lib/jetty-util-8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/server/lib/jetty-webapp-8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/server/lib/jetty-xml-8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/server/lib/servlet-api-3.0.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/example/example-DIH/solr/db/lib/derby-10.9.1.0.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/example/example-DIH/solr/db/lib/hsqldb-1.8.0.10.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build/core/classes/java:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/test-framework/lib/junit-4.10.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/test-framework/lib/randomizedtesting-runner-2.1.9.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/test-lib/cglib-nodep-2.2.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/test-lib/commons-collections-3.2.1.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/test-lib/easymock-3.0.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/test-lib/hadoop-common-2.2.0-tests.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/test-lib/hadoop-hdfs-2.2.0-tests.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/test-lib/javax.servlet-api-3.0.1.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/test-lib/jcl-over-slf4j-1.7.6.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/test-lib/jersey-core-1.9.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/test-lib/jetty-6.1.26.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/test-lib/jetty-util-6.1.26.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/test-lib/objenesis-1.2.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build/analysis/icu/lucene-analyzers-icu-6.0.0-SNAPSHOT.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/contrib/solr-analysis-extras/classes/java:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/contrib/analysis-extras/lib/icu4j-54.1.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-launcher.jar:/home/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jsch.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jai.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-regexp.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-netrexx.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-antlr.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit4.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bsf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-log4j.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bcel.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-resolver.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-swing.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-oro.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-xalan2.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jmf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jdepend.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-testutil.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-logging.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-net.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-javamail.jar:/usr/local/openjdk7/lib/tools.jar:/home/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.1.9.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/temp/junit4-J1-20141107_151248_360.events @/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/temp/junit4-J1-20141107_151248_360.suites
   [junit4] ERROR: JVM J1 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1501)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:133)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:964)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:961)
   [junit4] 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4] 	at java.lang.Thread.run(Thread.java:745)

BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:532: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:473: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:502: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1358: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:965: At least one slave process threw an exception, first: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.

Total time: 200 minutes 8 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #675
Archived 6 artifacts
Archive block size is 32768
Received 0 blocks and 1574545666 bytes
Compression is 0.0%
Took 5 min 15 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure