You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/11/14 06:15:36 UTC

[JENKINS] Lucene-Solr-trunk-Linux (32bit/ibm-j9-jdk7) - Build # 8307 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/8307/
Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}

6 tests failed.
REGRESSION:  org.apache.solr.cloud.BasicDistributedZk2Test.testDistribSearch

Error Message:
Expected content type application/octet-stream but got text/html;charset=ISO-8859-1. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 404 Can not find: /onenodecollectioncore/update</title> </head> <body> <h2>HTTP ERROR: 404</h2> <p>Problem accessing /onenodecollectioncore/update. Reason: <pre>    Can not find: /onenodecollectioncore/update</pre></p> <hr /><i><small>Powered by Jetty://</small></i>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected content type application/octet-stream but got text/html;charset=ISO-8859-1. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 404 Can not find: /onenodecollectioncore/update</title>
</head>
<body>
<h2>HTTP ERROR: 404</h2>
<p>Problem accessing /onenodecollectioncore/update. Reason:
<pre>    Can not find: /onenodecollectioncore/update</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
</body>
</html>

	at __randomizedtesting.SeedInfo.seed([FFE79EAEE9F482:81196986D9B694BE]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:428)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
	at org.apache.solr.cloud.BasicDistributedZk2Test.testNodeWithoutCollectionForwarding(BasicDistributedZk2Test.java:196)
	at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:88)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:613)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:780)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteReplicaTest

Error Message:
ERROR: SolrZkClient opens=15 closes=14

Stack Trace:
java.lang.AssertionError: ERROR: SolrZkClient opens=15 closes=14
	at __randomizedtesting.SeedInfo.seed([FFE79EAEE9F482]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:343)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:135)
	at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:613)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:700)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:780)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteReplicaTest

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.DeleteReplicaTest:     1) Thread[id=4568, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[FFE79EAEE9F482]-SendThread(localhost.localdomain:37445), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]         at java.lang.Thread.sleep(Native Method)         at java.lang.Thread.sleep(Thread.java:904)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)    2) Thread[id=4569, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[FFE79EAEE9F482]-EventThread, state=WAITING, group=TGRP-DeleteReplicaTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:197)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2054)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:453)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.DeleteReplicaTest: 
   1) Thread[id=4568, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[FFE79EAEE9F482]-SendThread(localhost.localdomain:37445), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
        at java.lang.Thread.sleep(Native Method)
        at java.lang.Thread.sleep(Thread.java:904)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   2) Thread[id=4569, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[FFE79EAEE9F482]-EventThread, state=WAITING, group=TGRP-DeleteReplicaTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:197)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2054)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:453)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
	at __randomizedtesting.SeedInfo.seed([FFE79EAEE9F482]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteReplicaTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=4568, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[FFE79EAEE9F482]-SendThread(localhost.localdomain:37445), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]         at java.lang.Thread.sleep(Native Method)         at java.lang.Thread.sleep(Thread.java:904)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=4568, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[FFE79EAEE9F482]-SendThread(localhost.localdomain:37445), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
        at java.lang.Thread.sleep(Native Method)
        at java.lang.Thread.sleep(Thread.java:904)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
	at __randomizedtesting.SeedInfo.seed([FFE79EAEE9F482]:0)


FAILED:  org.apache.solr.cloud.DeleteReplicaTest.testDistribSearch

Error Message:
Invalid replica : core_node2 in shard/collection : shard2/delLiveColl available replicas are core_node4

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Invalid replica : core_node2 in shard/collection : shard2/delLiveColl available replicas are core_node4
	at __randomizedtesting.SeedInfo.seed([FFE79EAEE9F482:81196986D9B694BE]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:464)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:268)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:640)
	at org.apache.solr.cloud.DeleteReplicaTest.removeAndWaitForReplicaGone(DeleteReplicaTest.java:154)
	at org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:141)
	at org.apache.solr.cloud.DeleteReplicaTest.doTest(DeleteReplicaTest.java:115)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:613)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:780)


FAILED:  org.apache.solr.update.AutoCommitTest.testMaxTime

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
	at __randomizedtesting.SeedInfo.seed([FFE79EAEE9F482:9A0B9A7C307368BE]:0)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:637)
	at org.apache.solr.update.AutoCommitTest.testMaxTime(AutoCommitTest.java:227)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:613)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:780)
Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=0]
	xml response was: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">1125</int></lst><result name="response" numFound="1" start="0"><doc><int name="id">529</int><int name="range_facet_si">529</int><arr name="range_facet_l"><long>529</long></arr><arr name="range_facet_sl"><long>529</long></arr><arr name="field_t"><str>what's inside?</str></arr><arr name="subject"><str>info</str></arr><int name="intDefault">42</int><date name="timestamp">2013-11-14T05:06:26.128Z</date><arr name="multiDefault"><str>muLti-Default</str></arr></doc></result>
</response>

	request was:q=id:529&start=0&qt=standard&version=2.2&rows=20
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:630)
	... 41 more




Build Log:
[...truncated 9932 lines...]
   [junit4] Suite: org.apache.solr.cloud.BasicDistributedZk2Test
   [junit4]   2> 42788 T221 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 42793 T221 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1384404777234
   [junit4]   2> 42796 T221 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 42798 T222 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 42898 T221 oasc.ZkTestServer.run start zk server on port:56251
   [junit4]   2> 42938 T221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 43050 T230 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d739cd07 name:ZooKeeperConnection Watcher:127.0.0.1:56251 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 43050 T221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 43051 T221 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 43068 T221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 43070 T232 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ee987f56 name:ZooKeeperConnection Watcher:127.0.0.1:56251/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 43070 T221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 43072 T221 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 43076 T221 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 43081 T221 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 43085 T221 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 43088 T221 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 43093 T221 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 43099 T221 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 43099 T221 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 43104 T221 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 43105 T221 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 43108 T221 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 43109 T221 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 43113 T221 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 43114 T221 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 43118 T221 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 43118 T221 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 43122 T221 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 43122 T221 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 43126 T221 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 43127 T221 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 43130 T221 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 43131 T221 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 43135 T221 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 43136 T221 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 43139 T221 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 43140 T221 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 43223 T221 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 43228 T221 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:33200
   [junit4]   2> 43228 T221 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 43229 T221 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 43230 T221 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1384404777584
   [junit4]   2> 43230 T221 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1384404777584/'
   [junit4]   2> 43239 T221 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1384404777584/solr.xml
   [junit4]   2> 43270 T221 oasc.CoreContainer.<init> New CoreContainer 207498361
   [junit4]   2> 43271 T221 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1384404777584/]
   [junit4]   2> 43272 T221 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 43273 T221 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 43273 T221 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 43274 T221 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 43274 T221 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 43275 T221 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 43276 T221 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 43276 T221 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 43277 T221 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 43278 T221 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 43278 T221 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 43279 T221 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 43280 T221 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56251/solr
   [junit4]   2> 43285 T221 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 43286 T221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 43288 T243 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@596c5b0e name:ZooKeeperConnection Watcher:127.0.0.1:56251 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 43288 T221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 43293 T221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 43295 T245 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f96bf039 name:ZooKeeperConnection Watcher:127.0.0.1:56251/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 43295 T221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 43299 T221 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 43305 T221 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 43311 T221 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 43314 T221 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33200_
   [junit4]   2> 43317 T221 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33200_
   [junit4]   2> 43322 T221 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 43331 T221 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 43335 T221 oasc.Overseer.start Overseer (id=90728351483691011-127.0.0.1:33200_-n_0000000000) starting
   [junit4]   2> 43342 T221 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 43351 T247 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 43351 T221 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 43356 T221 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 43358 T221 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 43364 T246 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 43367 T248 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 43367 T248 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 43369 T248 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 44873 T246 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 44875 T246 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:33200",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:33200_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 44876 T246 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 44880 T246 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 44891 T245 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 45369 T248 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 45370 T248 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1384404777584/collection1
   [junit4]   2> 45371 T248 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 45373 T248 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 45374 T248 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 45383 T248 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1384404777584/collection1/'
   [junit4]   2> 45385 T248 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1384404777584/collection1/lib/classes/' to classloader
   [junit4]   2> 45386 T248 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1384404777584/collection1/lib/README' to classloader
   [junit4]   2> 45436 T248 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 45470 T248 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 45472 T248 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 45479 T248 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 46101 T248 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 46107 T248 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 46110 T248 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 46133 T248 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 46139 T248 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 46145 T248 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 46147 T248 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 46148 T248 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 46148 T248 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 46150 T248 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 46150 T248 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 46151 T248 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 46151 T248 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1384404777584/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1384404777234/control/data/
   [junit4]   2> 46152 T248 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@f2ea651f
   [junit4]   2> 46156 T248 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1384404777234/control/data
   [junit4]   2> 46157 T248 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1384404777234/control/data/index/
   [junit4]   2> 46158 T248 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1384404777234/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 46159 T248 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1384404777234/control/data/index
   [junit4]   2> 46159 T248 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=42, maxMergedSegmentMB=88.8486328125, floorSegmentMB=2.1484375, forceMergeDeletesPctAllowed=7.508696075230075, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 46160 T248 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@e0741bf3 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@fa801abe),segFN=segments_1,generation=1}
   [junit4]   2> 46161 T248 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 46170 T248 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 46175 T248 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 46175 T248 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 46176 T248 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 46176 T248 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 46176 T248 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 46177 T248 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 46177 T248 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 46178 T248 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 46180 T248 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 46182 T248 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 46182 T248 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 46183 T248 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 46184 T248 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 46185 T248 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 46186 T248 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 46189 T248 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 46189 T248 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 46191 T248 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=26, maxMergedSegmentMB=55.7783203125, floorSegmentMB=0.65234375, forceMergeDeletesPctAllowed=9.800313300705458, segmentsPerTier=30.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 46191 T248 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@e0741bf3 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@fa801abe),segFN=segments_1,generation=1}
   [junit4]   2> 46192 T248 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 46192 T248 oass.SolrIndexSearcher.<init> Opening Searcher@331029b5 main
   [junit4]   2> 46210 T249 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@331029b5 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 46219 T248 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 46220 T248 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:33200 collection:control_collection shard:shard1
   [junit4]   2> 46223 T248 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 46233 T248 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 46236 T248 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 46236 T248 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 46237 T248 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:33200/collection1/
   [junit4]   2> 46238 T248 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 46238 T248 oasc.SyncStrategy.syncToMe http://127.0.0.1:33200/collection1/ has no replicas
   [junit4]   2> 46238 T248 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:33200/collection1/ shard1
   [junit4]   2> 46239 T248 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 46397 T246 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 46408 T245 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 46448 T248 oasc.ZkController.register We are http://127.0.0.1:33200/collection1/ and leader is http://127.0.0.1:33200/collection1/
   [junit4]   2> 46448 T248 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:33200
   [junit4]   2> 46448 T248 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 46449 T248 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 46449 T248 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 46451 T248 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 46453 T221 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 46454 T221 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 46458 T221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 46460 T252 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5889a6d9 name:ZooKeeperConnection Watcher:127.0.0.1:56251/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 46461 T221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 46462 T221 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 46469 T221 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 46579 T221 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 46582 T221 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59992
   [junit4]   2> 46583 T221 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 46583 T221 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 46584 T221 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1384404780910
   [junit4]   2> 46585 T221 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1384404780910/'
   [junit4]   2> 46594 T221 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1384404780910/solr.xml
   [junit4]   2> 46632 T221 oasc.CoreContainer.<init> New CoreContainer -1563579819
   [junit4]   2> 46633 T221 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1384404780910/]
   [junit4]   2> 46634 T221 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 46635 T221 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 46635 T221 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 46636 T221 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 46636 T221 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 46637 T221 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 46638 T221 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 46638 T221 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 46639 T221 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 46640 T221 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 46640 T221 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 46641 T221 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 46642 T221 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56251/solr
   [junit4]   2> 46642 T221 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 46644 T221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 46646 T263 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e95412df name:ZooKeeperConnection Watcher:127.0.0.1:56251 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 46646 T221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 46649 T221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 46651 T265 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@db68049c name:ZooKeeperConnection Watcher:127.0.0.1:56251/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 46651 T221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 46657 T221 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 47663 T221 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59992_
   [junit4]   2> 47665 T221 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59992_
   [junit4]   2> 47670 T252 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 47670 T265 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 47670 T245 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 47680 T266 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 47680 T266 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 47684 T266 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 47922 T246 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 47925 T246 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:33200",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:33200_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 47934 T246 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:59992",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:59992_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 47935 T246 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 47936 T246 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 47946 T245 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 47946 T252 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 47946 T265 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 48684 T266 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 48685 T266 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1384404780910/collection1
   [junit4]   2> 48686 T266 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 48688 T266 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 48689 T266 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 48692 T266 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1384404780910/collection1/'
   [junit4]   2> 48693 T266 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1384404780910/collection1/lib/classes/' to classloader
   [junit4]   2> 48694 T266 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1384404780910/collection1/lib/README' to classloader
   [junit4]   2> 48759 T266 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 48791 T266 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 48795 T266 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 48802 T266 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 49128 T266 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 49133 T266 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 49136 T266 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 49155 T266 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 49163 T266 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 49177 T266 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 49180 T266 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 49182 T266 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 49183 T266 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 49186 T266 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 49187 T266 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 49189 T266 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 49190 T266 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1384404780910/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1384404777234/jetty1/
   [junit4]   2> 49191 T266 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@f2ea651f
   [junit4]   2> 49196 T266 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1384404777234/jetty1
   [junit4]   2> 49197 T266 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1384404777234/jetty1/index/
   [junit4]   2> 49198 T266 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1384404777234/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 49198 T266 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1384404777234/jetty1/index
   [junit4]   2> 49199 T266 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=42, maxMergedSegmentMB=88.8486328125, floorSegmentMB=2.1484375, forceMergeDeletesPctAllowed=7.508696075230075, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 49200 T266 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@a3cb3c65 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@dad8279b),segFN=segments_1,generation=1}
   [junit4]   2> 49201 T266 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 49213 T266 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 49214 T266 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 49214 T266 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 49215 T266 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 49215 T266 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 49215 T266 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 49216 T266 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 49216 T266 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 49216 T266 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 49218 T266 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 49219 T266 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 49220 T266 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 49220 T266 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 49222 T266 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 49223 T266 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 49224 T266 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 49226 T266 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 49227 T266 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 49228 T266 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=26, maxMergedSegmentMB=55.7783203125, floorSegmentMB=0.65234375, forceMergeDeletesPctAllowed=9.800313300705458, segmentsPerTier=30.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 49229 T266 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@a3cb3c65 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@dad8279b),segFN=segments_1,generation=1}
   [junit4]   2> 49229 T266 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 49230 T266 oass.SolrIndexSearcher.<init> Opening Searcher@bbb848e2 main
   [junit4]   2> 49246 T267 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@bbb848e2 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 49255 T266 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 49256 T266 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:59992 collection:collection1 shard:shard1
   [junit4]   2> 49257 T266 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 49267 T266 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 49270 T266 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 49270 T266 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 49270 T266 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:59992/collection1/
   [junit4]   2> 49271 T266 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 49271 T266 oasc.SyncStrategy.syncToMe http://127.0.0.1:59992/collection1/ has no replicas
   [junit4]   2> 49271 T266 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:59992/collection1/ shard1
   [junit4]   2> 49272 T266 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 49458 T246 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 49467 T245 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 49467 T252 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 49467 T265 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 49481 T266 oasc.ZkController.register We are http://127.0.0.1:59992/collection1/ and leader is http://127.0.0.1:59992/collection1/
   [junit4]   2> 49481 T266 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:59992
   [junit4]   2> 49481 T266 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 49482 T266 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 49482 T266 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 49485 T266 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 49486 T221 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 49487 T221 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 49576 T221 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 49579 T221 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:46497
   [junit4]   2> 49580 T221 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 49580 T221 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 49581 T221 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1384404783928
   [junit4]   2> 49582 T221 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1384404783928/'
   [junit4]   2> 49590 T221 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1384404783928/solr.xml
   [junit4]   2> 49622 T221 oasc.CoreContainer.<init> New CoreContainer 309209710
   [junit4]   2> 49622 T221 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1384404783928/]
   [junit4]   2> 49624 T221 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 49624 T221 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 49625 T221 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 49625 T221 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 49626 T221 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 49627 T221 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 49627 T221 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 49628 T221 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 49628 T221 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 49630 T221 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 49630 T221 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 49631 T221 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 49632 T221 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56251/solr
   [junit4]   2> 49633 T221 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 49634 T221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 49637 T279 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1cfdbd8 name:ZooKeeperConnection Watcher:127.0.0.1:56251 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 49638 T221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 49642 T221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 49645 T281 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@eba6fb4c name:ZooKeeperConnection Watcher:127.0.0.1:56251/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 49646 T221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 49654 T221 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 50661 T221 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46497_
   [junit4]   2> 50663 T221 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46497_
   [junit4]   2> 50667 T281 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 50667 T252 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 50669 T265 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 50669 T245 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 50678 T282 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 50678 T282 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 50680 T282 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 50973 T246 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 50974 T246 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:59992",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:59992_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 50979 T246 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:46497",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:46497_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 50979 T246 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 50980 T246 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 50990 T252 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 50990 T265 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 50990 T281 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 50991 T245 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 51680 T282 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 51681 T282 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1384404783928/collection1
   [junit4]   2> 51681 T282 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 51683 T282 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 51683 T282 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 51685 T282 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1384404783928/collection1/'
   [junit4]   2> 51685 T282 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1384404783928/collection1/lib/classes/' to classloader
   [junit4]   2> 51686 T282 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1384404783928/collection1/lib/README' to classloader
   [junit4]   2> 51735 T282 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 51794 T282 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 51798 T282 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 51806 T282 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 52273 T282 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 52280 T282 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 52282 T282 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 52304 T282 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 52310 T282 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 52317 T282 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 52318 T282 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 52319 T282 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 52320 T282 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 52321 T282 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 52322 T282 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 52323 T282 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 52323 T282 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1384404783928/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1384404777234/jetty2/
   [junit4]   2> 52324 T282 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@f2ea651f
   [junit4]   2> 52327 T282 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1384404777234/jetty2
   [junit4]   2> 52327 T282 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1384404777234/jetty2/index/
   [junit4]   2> 52328 T282 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1384404777234/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 52329 T282 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1384404777234/jetty2/index
   [junit4]   2> 52330 T282 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=42, maxMergedSegmentMB=88.8486328125, floorSegmentMB=2.1484375, forceMergeDeletesPctAllowed=7.508696075230075, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 52331 T282 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@81cd5659 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@335053db),segFN=segments_1,generation=1}
   [junit4]   2> 52331 T282 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 52341 T282 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 52342 T282 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 52342 T282 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 52342 T282 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 52343 T282 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 52343 T282 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 52344 T282 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 52344 T282 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 52345 T282 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 52346 T282 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 52348 T282 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 52348 T282 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 52348 T282 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 52350 T282 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 52350 T282 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 52352 T282 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 52354 T282 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 52355 T282 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 52356 T282 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=26, maxMergedSegmentMB=55.7783203125, floorSegmentMB=0.65234375, forceMergeDeletesPctAllowed=9.800313300705458, segmentsPerTier=30.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 52357 T282 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@81cd5659 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@335053db),segFN=segments_1,generation=1}
   [junit4]   2> 52357 T282 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 52358 T282 oass.SolrIndexSearcher.<init> Opening Searcher@8006e810 main
   [junit4]   2> 52368 T283 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@8006e810 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 52374 T282 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 52374 T282 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:46497 collection:collection1 shard:shard2
   [junit4]   2> 52376 T282 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 52383 T282 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 52385 T282 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 52386 T282 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 52386 T282 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:46497/collection1/
   [junit4]   2> 52386 T282 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 52387 T282 oasc.SyncStrategy.syncToMe http://127.0.0.1:46497/collection1/ has no replicas
   [junit4]   2> 52387 T282 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:46497/collection1/ shard2
   [junit4]   2> 52387 T282 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 52499 T246 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 52510 T281 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 52510 T265 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 52510 T245 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 52510 T252 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 52544 T282 oasc.ZkController.register We are http://127.0.0.1:46497/collection1/ and leader is http://127.0.0.1:46497/collection1/
   [junit4]   2> 52544 T282 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:46497
   [junit4]   2> 52544 T282 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 52545 T282 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 52545 T282 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 52547 T282 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 52550 T221 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 52550 T221 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 52646 T221 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 52649 T221 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:38861
   [junit4]   2> 52650 T221 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 52651 T221 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 52651 T221 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1384404786992
   [junit4]   2> 52652 T221 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1384404786992/'
   [junit4]   2> 52661 T221 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1384404786992/solr.xml
   [junit4]   2> 52695 T221 oasc.CoreContainer.<init> New CoreContainer 1224192357
   [junit4]   2> 52695 T221 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1384404786992/]
   [junit4]   2> 52697 T221 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 52697 T221 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 52698 T221 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 52699 T221 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 52699 T221 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 52700 T221 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 52701 T221 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 52702 T221 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 52702 T221 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 52704 T221 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 52704 T221 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 52705 T221 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 52706 T221 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56251/solr
   [junit4]   2> 52706 T221 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 52708 T221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 52711 T295 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e59943ca name:ZooKeeperConnection Watcher:127.0.0.1:56251 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 52712 T221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 52715 T221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 52718 T297 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a2dec120 name:ZooKeeperConnection Watcher:127.0.0.1:56251/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 52719 T221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 52726 T221 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 53731 T221 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38861_
   [junit4]   2> 53733 T221 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38861_
   [junit4]   2> 53737 T265 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 53737 T297 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 53737 T245 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 53737 T252 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 53737 T281 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 53754 T298 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 53754 T298 oasc.ZkController.publish numShards not found on descriptor - readi

[...truncated too long message...]

ntCnxn.java:984)
   [junit4]   2> 
   [junit4]   2> 1247946 T4568 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at java.lang.Thread.sleep(Thread.java:904)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 1247948 T4568 oaz.ClientCnxn$SendThread.run WARN Session 0x14255054b51000e for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:706)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 103281 T4481 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
   [junit4]   2> 	   1) Thread[id=4568, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[FFE79EAEE9F482]-SendThread(localhost.localdomain:37445), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]   2> 	        at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	        at java.lang.Thread.sleep(Thread.java:904)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   [junit4]   2> NOTE: test params are: codec=DummyCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=DUMMY, chunkSize=487), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=DUMMY, chunkSize=487)), sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=es_BO, timezone=Etc/GMT-2
   [junit4]   2> NOTE: Linux 3.8.0-32-generic x86/IBM Corporation 1.7.0 (32-bit)/cpus=8,threads=3,free=93722440,total=245497856
   [junit4]   2> NOTE: All tests run in this JVM: [QueryElevationComponentTest, ShowFileRequestHandlerTest, TestPHPSerializedResponseWriter, TestSolrQueryParser, TestFieldResource, SampleTest, TestElisionMultitermQuery, UnloadDistributedZkTest, TestRandomMergePolicy, MBeansHandlerTest, TestDynamicFieldCollectionResource, SOLR749Test, TestSolrIndexConfig, TestSolrXml, LegacyHTMLStripCharFilterTest, TestLFUCache, TestConfig, SimpleFacetsTest, TestJmxMonitoredMap, LeaderElectionTest, SolrTestCaseJ4Test, DirectUpdateHandlerOptimizeTest, TestBadConfig, ResourceLoaderTest, SolrIndexSplitterTest, TestSuggestSpellingConverter, CustomCollectionTest, TestValueSourceCache, ResponseLogComponentTest, LoggingHandlerTest, TestSurroundQueryParser, RecoveryZkTest, SynonymTokenizerTest, TestImplicitCoreProperties, TestCopyFieldCollectionResource, URLClassifyProcessorTest, TestManagedSchema, TestPseudoReturnFields, TestCollapseQParserPlugin, SearchHandlerTest, TermVectorComponentTest, QueryParsingTest, TestBM25SimilarityFactory, CopyFieldTest, CoreMergeIndexesAdminHandlerTest, TestCharFilters, PolyFieldTest, AliasIntegrationTest, SystemInfoHandlerTest, ScriptEngineTest, ClusterStateTest, AddBlockUpdateTest, TestReversedWildcardFilterFactory, SuggesterTest, FieldMutatingUpdateProcessorTest, BadComponentTest, CacheHeaderTest, CachingDirectoryFactoryTest, DistributedTermsComponentTest, BinaryUpdateRequestHandlerTest, UpdateRequestProcessorFactoryTest, TestLMJelinekMercerSimilarityFactory, SpellPossibilityIteratorTest, SliceStateUpdateTest, TestSerializedLuceneMatchVersion, SolrInfoMBeanTest, HdfsLockFactoryTest, ZkCLITest, TestDynamicFieldResource, CoreAdminHandlerTest, TestWriterPerf, MoreLikeThisHandlerTest, FileBasedSpellCheckerTest, TestCoreContainer, TestDocumentBuilder, TestLRUCache, DocumentBuilderTest, SuggesterFSTTest, IndexSchemaTest, CSVRequestHandlerTest, HdfsDirectoryTest, CollectionsAPIDistributedZkTest, ShardRoutingTest, FullSolrCloudDistribCmdsTest, TestReplicationHandler, ClusterStateUpdateTest, TestZkChroot, TestRandomDVFaceting, TestFaceting, TestHashPartitioner, ZkControllerTest, TestRealTimeGet, TestMultiCoreConfBootstrap, TestReload, TestStressVersions, SolrCoreTest, StatsComponentTest, TestGroupingSearch, TestFunctionQuery, BasicFunctionalityTest, OverseerCollectionProcessorTest, HighlighterTest, SoftAutoCommitTest, CurrencyFieldOpenExchangeTest, DistributedQueryElevationComponentTest, TestCoreDiscovery, AnalysisAfterCoreReloadTest, SignatureUpdateProcessorFactoryTest, SpellCheckCollatorTest, SpatialFilterTest, SchemaVersionSpecificBehaviorTest, TestAtomicUpdateErrorCases, DefaultValueUpdateProcessorTest, SortByFunctionTest, DocValuesMultiTest, TestSolrDeletionPolicy1, LukeRequestHandlerTest, TermsComponentTest, RequestHandlersTest, FieldAnalysisRequestHandlerTest, IndexSchemaRuntimeFieldTest, TestCollationField, TestCSVResponseWriter, PingRequestHandlerTest, TestBinaryResponseWriter, UpdateParamsTest, SolrIndexConfigTest, TestMergePolicyConfig, TestSolrDeletionPolicy2, MultiTermTest, TestDocSet, TestBinaryField, TestSearchPerf, MinimalSchemaTest, OutputWriterTest, ExternalFileFieldSortTest, TestPostingsSolrHighlighter, DirectSolrConnectionTest, NotRequiredUniqueKeyTest, TestLuceneMatchVersion, TestCodecSupport, EchoParamsTest, TestSweetSpotSimilarityFactory, TestPerFieldSimilarity, TestLMDirichletSimilarityFactory, TestIBSimilarityFactory, TestDefaultSimilarityFactory, TimeZoneUtilsTest, TestFastLRUCache, DateMathParserTest, TestSolrXMLSerializer, TestSystemIdResolver, SpellingQueryConverterTest, DOMUtilTest, TestSolrJ, ZkNodePropsTest, SliceStateTest, UUIDFieldTest, FileUtilsTest, CircularListTest, TestRTGBase, DeleteInactiveReplicaTest, DeleteReplicaTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DeleteReplicaTest -Dtests.seed=FFE79EAEE9F482 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_BO -Dtests.timezone=Etc/GMT-2 -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J0 | DeleteReplicaTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ERROR: SolrZkClient opens=15 closes=14
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([FFE79EAEE9F482]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:343)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:135)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:780)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.DeleteReplicaTest: 
   [junit4]    >    1) Thread[id=4568, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[FFE79EAEE9F482]-SendThread(localhost.localdomain:37445), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at java.lang.Thread.sleep(Thread.java:904)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   [junit4]    >    2) Thread[id=4569, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[FFE79EAEE9F482]-EventThread, state=WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:197)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2054)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:453)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([FFE79EAEE9F482]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=4568, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[FFE79EAEE9F482]-SendThread(localhost.localdomain:37445), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at java.lang.Thread.sleep(Thread.java:904)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([FFE79EAEE9F482]:0)
   [junit4] Completed on J0 in 103.33s, 1 test, 1 failure, 3 errors <<< FAILURES!

[...truncated 129 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:423: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:416: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:489: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1272: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:908: There were test failures: 335 suites, 1473 tests, 3 suite-level errors, 3 errors, 123 ignored (6 assumptions)

Total time: 48 minutes 50 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure