You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/02/16 16:40:51 UTC

[JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_60-ea-b04) - Build # 3786 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/3786/
Java: 64bit/jdk1.7.0_60-ea-b04 -XX:+UseCompressedOops -XX:+UseG1GC

4 tests failed.
FAILED:  org.apache.solr.TestDistributedGrouping.testDistribSearch

Error Message:
Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 Server Error</title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /hs_/bm/select. Reason: <pre>    Server Error</pre></p> <hr /><i><small>Powered by Jetty://</small></i>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 Server Error</title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /hs_/bm/select. Reason:
<pre>    Server Error</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
</body>
</html>

	at __randomizedtesting.SeedInfo.seed([9FBBAC6CBCF702C4:1E5D2274CBA862F8]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:459)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:90)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.BaseDistributedSearchTestCase.queryServer(BaseDistributedSearchTestCase.java:515)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:563)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:545)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:524)
	at org.apache.solr.TestDistributedGrouping.doTest(TestDistributedGrouping.java:156)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:876)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.solr.TestDistributedSearch.testDistribSearch

Error Message:
Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 Server Error</title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /select. Reason: <pre>    Server Error</pre></p> <hr /><i><small>Powered by Jetty://</small></i>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 Server Error</title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /select. Reason:
<pre>    Server Error</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
</body>
</html>

	at __randomizedtesting.SeedInfo.seed([9FBBAC6CBCF702C4:1E5D2274CBA862F8]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:459)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:90)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.BaseDistributedSearchTestCase.queryServer(BaseDistributedSearchTestCase.java:515)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:563)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:545)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:524)
	at org.apache.solr.TestDistributedSearch.doTest(TestDistributedSearch.java:177)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:876)
	at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.solr.cloud.BasicDistributedZk2Test.testDistribSearch

Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:49288/pc/e/collection1, http://127.0.0.1:49278/pc/e/collection1, http://127.0.0.1:49260/pc/e/collection1, http://127.0.0.1:49269/pc/e/collection1]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:49288/pc/e/collection1, http://127.0.0.1:49278/pc/e/collection1, http://127.0.0.1:49260/pc/e/collection1, http://127.0.0.1:49269/pc/e/collection1]
	at __randomizedtesting.SeedInfo.seed([9FBBAC6CBCF702C4:1E5D2274CBA862F8]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:362)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:635)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:90)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.queryServer(AbstractFullDistribZkTestBase.java:1356)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:563)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:545)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:524)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkQueries(AbstractFullDistribZkTestBase.java:797)
	at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:101)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 Server Error</title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /pc/e/collection1/select. Reason:
<pre>    Server Error</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
</body>
</html>

	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:459)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:293)
	... 49 more


FAILED:  org.apache.solr.cloud.BasicDistributedZkTest.testDistribSearch

Error Message:
No live SolrServers available to handle this request:[https://127.0.0.1:50560/collection1, https://127.0.0.1:50587/collection1, https://127.0.0.1:50569/collection1, https://127.0.0.1:50578/collection1]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:50560/collection1, https://127.0.0.1:50587/collection1, https://127.0.0.1:50569/collection1, https://127.0.0.1:50578/collection1]
	at __randomizedtesting.SeedInfo.seed([9FBBAC6CBCF702C4:1E5D2274CBA862F8]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:362)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:635)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:90)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.cloud.BasicDistributedZkTest.queryServer(BasicDistributedZkTest.java:1176)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:563)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:545)
	at org.apache.solr.cloud.BasicDistributedZkTest.doTest(BasicDistributedZkTest.java:264)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 Server Error</title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /collection1/select. Reason:
<pre>    Server Error</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
</body>
</html>

	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:459)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:293)
	... 48 more




Build Log:
[...truncated 9871 lines...]
   [junit4] Suite: org.apache.solr.cloud.BasicDistributedZkTest
   [junit4]   2> 72637 T60 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 72638 T60 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 72645 T60 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\solrtest-BasicDistributedZkTest-1392562094464
   [junit4]   2> 72653 T60 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 72692 T61 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 72882 T60 oasc.ZkTestServer.run start zk server on port:50537
   [junit4]   2> 73069 T60 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 73104 T62 oazs.NIOServerCnxn.doIO WARN Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
   [junit4]   2> 74361 T67 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e217f8f name:ZooKeeperConnection Watcher:127.0.0.1:50537 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 74362 T60 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 74366 T60 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 74468 T60 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 74481 T69 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d053579 name:ZooKeeperConnection Watcher:127.0.0.1:50537/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 74482 T60 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 74491 T60 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 74502 T60 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 74511 T60 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 74520 T60 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 74535 T60 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 74537 T60 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 74554 T60 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 74555 T60 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 74569 T60 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 74570 T60 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 74581 T60 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 74581 T60 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 74592 T60 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 74593 T60 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 74601 T60 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 74601 T60 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 74611 T60 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 74611 T60 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 74621 T60 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 74622 T60 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 74631 T60 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 74632 T60 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 74640 T60 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 74641 T60 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 74648 T60 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 74649 T60 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 74671 T60 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 74677 T71 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@43e6a74e name:ZooKeeperConnection Watcher:127.0.0.1:50537/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 74678 T60 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 74689 T62 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1443b2b42b90002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 75625 T60 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 76104 T60 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 76152 T60 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:50548
   [junit4]   2> 76246 T60 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 76247 T60 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 76247 T60 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: .\org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1392562096502
   [junit4]   2> 76248 T60 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '.\org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1392562096502\'
   [junit4]   2> 76284 T60 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1392562096502\solr.xml
   [junit4]   2> 76353 T60 oasc.CoreContainer.<init> New CoreContainer 147555268
   [junit4]   2> 76354 T60 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=.\org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1392562096502\]
   [junit4]   2> 76414 T60 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 76414 T60 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 76414 T60 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 76415 T60 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 76415 T60 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 76416 T60 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 76416 T60 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 76416 T60 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 76417 T60 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 76439 T60 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 76439 T60 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 76440 T60 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 76441 T60 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50537/solr
   [junit4]   2> 76460 T60 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 76464 T60 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 76480 T83 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4c18df89 name:ZooKeeperConnection Watcher:127.0.0.1:50537 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 76481 T60 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 76502 T60 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 76505 T85 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@129dc9b8 name:ZooKeeperConnection Watcher:127.0.0.1:50537/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 76506 T60 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 76520 T60 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 76541 T60 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 76570 T60 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 76575 T60 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50548_
   [junit4]   2> 76583 T60 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50548_
   [junit4]   2> 76593 T60 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 76600 T60 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 76626 T60 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:50548_
   [junit4]   2> 76627 T60 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 76634 T60 oasc.Overseer.start Overseer (id=91262949443305476-127.0.0.1:50548_-n_0000000000) starting
   [junit4]   2> 76660 T60 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 76684 T87 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 76690 T87 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 76691 T60 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 76702 T60 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 76708 T60 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 76725 T86 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 76738 T88 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 76739 T88 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 76747 T85 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 76747 T88 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 76757 T86 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 76759 T86 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50548",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50548_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 76759 T86 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 76786 T86 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 76796 T85 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 76808 T85 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 77694 T88 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 77695 T88 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: .\org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1392562096502\collection1
   [junit4]   2> 77695 T88 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 77699 T88 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 77699 T88 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 77704 T88 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 77706 T88 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '.\org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1392562096502\collection1\'
   [junit4]   2> 77713 T88 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1392562096502/collection1/lib/classes/' to classloader
   [junit4]   2> 77714 T88 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1392562096502/collection1/lib/README' to classloader
   [junit4]   2> 77808 T88 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 77852 T88 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 77856 T88 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 77889 T88 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 78366 T88 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 78379 T88 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 78382 T88 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 78444 T88 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 78451 T88 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 78458 T88 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 78461 T88 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 78461 T88 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 78461 T88 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 78464 T88 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 78464 T88 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 78464 T88 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 78465 T88 oasc.SolrCore.<init> [collection1] Opening new SolrCore at .\org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1392562096502\collection1\, dataDir=.\org.apache.solr.cloud.BasicDistributedZkTest-1392562094463/control/data\
   [junit4]   2> 78466 T88 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@53e3223e
   [junit4]   2> 78473 T88 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.BasicDistributedZkTest-1392562094463/control/data\
   [junit4]   2> 78473 T88 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=.\org.apache.solr.cloud.BasicDistributedZkTest-1392562094463/control/data\index/
   [junit4]   2> 78474 T88 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '.\org.apache.solr.cloud.BasicDistributedZkTest-1392562094463\control\data\index' doesn't exist. Creating new index...
   [junit4]   2> 78474 T88 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.BasicDistributedZkTest-1392562094463/control/data\index
   [junit4]   2> 78485 T88 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=697186585, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 78486 T88 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@5f1b98ca lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@694f67b4),segFN=segments_1,generation=1}
   [junit4]   2> 78486 T88 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 78501 T88 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 78546 T88 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 78546 T88 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 78546 T88 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 78546 T88 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 78547 T88 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 78547 T88 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 78548 T88 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 78548 T88 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 78549 T88 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 78550 T88 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 78550 T88 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 78551 T88 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 78551 T88 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 78553 T88 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 78553 T88 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 78554 T88 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 78561 T88 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 78568 T88 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 78568 T88 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 78569 T88 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=48, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=2.0361328125, floorSegmentMB=0.515625, forceMergeDeletesPctAllowed=6.886370762905878, segmentsPerTier=22.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 78570 T88 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@5f1b98ca lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@694f67b4),segFN=segments_1,generation=1}
   [junit4]   2> 78570 T88 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 78570 T88 oass.SolrIndexSearcher.<init> Opening Searcher@eaf60a7[collection1] main
   [junit4]   2> 78579 T89 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@eaf60a7[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 78583 T88 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 78586 T60 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 78586 T60 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 78586 T92 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:50548 collection:control_collection shard:shard1
   [junit4]   2> 78604 T92 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 78625 T92 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 78631 T85 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 78644 T92 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 78644 T92 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C83 name=collection1 org.apache.solr.core.SolrCore@14722a22 url=https://127.0.0.1:50548/collection1 node=127.0.0.1:50548_ C83_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:50548, core=collection1, node_name=127.0.0.1:50548_}
   [junit4]   2> 78644 T92 C83 P50548 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:50548/collection1/
   [junit4]   2> 78645 T92 C83 P50548 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 78645 T92 C83 P50548 oasc.SyncStrategy.syncToMe https://127.0.0.1:50548/collection1/ has no replicas
   [junit4]   2> 78645 T92 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:50548/collection1/ shard1
   [junit4]   2> 78646 T92 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 78651 T86 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 78652 T60 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 78657 T94 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f938429 name:ZooKeeperConnection Watcher:127.0.0.1:50537/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 78657 T60 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 78663 T60 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 78664 T85 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 78666 T85 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 78667 T94 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 78677 T60 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 78683 T86 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 78692 T85 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 78799 T85 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 78799 T94 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 78819 T92 oasc.ZkController.register We are https://127.0.0.1:50548/collection1/ and leader is https://127.0.0.1:50548/collection1/
   [junit4]   2> 78819 T92 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:50548
   [junit4]   2> 78819 T92 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 78819 T92 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 78819 T92 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 78823 T85 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 78823 T85 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 78823 T85 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 78827 T92 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 78830 T86 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 78832 T86 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50548",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50548_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 78840 T85 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 78949 T94 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 78951 T85 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 79217 T60 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 79218 T60 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 79231 T60 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 79236 T60 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:50560
   [junit4]   2> 79240 T60 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 79240 T60 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 79240 T60 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1392562100495
   [junit4]   2> 79240 T60 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1392562100495\'
   [junit4]   2> 79275 T60 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1392562100495\solr.xml
   [junit4]   2> 79342 T60 oasc.CoreContainer.<init> New CoreContainer 2042247328
   [junit4]   2> 79343 T60 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1392562100495\]
   [junit4]   2> 79346 T60 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 79346 T60 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 79346 T60 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 79346 T60 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 79348 T60 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 79348 T60 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 79348 T60 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 79348 T60 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 79349 T60 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 79372 T60 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 79376 T60 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 79376 T60 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 79376 T60 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50537/solr
   [junit4]   2> 79377 T60 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 79380 T60 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 79394 T105 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@61628ac5 name:ZooKeeperConnection Watcher:127.0.0.1:50537 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 79395 T60 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 79400 T62 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1443b2b42b90006, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 79403 T60 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 79408 T107 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c2e4720 name:ZooKeeperConnection Watcher:127.0.0.1:50537/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 79409 T60 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 79424 T60 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 80473 T60 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50560_
   [junit4]   2> 80477 T60 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50560_
   [junit4]   2> 80484 T94 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 80485 T85 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 80486 T107 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 80505 T108 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 80505 T108 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 80508 T85 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 80508 T85 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 80509 T108 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 80509 T85 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 80514 T86 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 80516 T86 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50560",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50560_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 80516 T86 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 80517 T86 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 80527 T85 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 80533 T94 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 80533 T107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 80533 T85 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 81550 T108 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 81550 T108 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1392562100495\collection1
   [junit4]   2> 81550 T108 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 81552 T108 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 81552 T108 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 81556 T108 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 81556 T108 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1392562100495\collection1\'
   [junit4]   2> 81559 T108 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1392562100495/collection1/lib/classes/' to classloader
   [junit4]   2> 81559 T108 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1392562100495/collection1/lib/README' to classloader
   [junit4]   2> 81625 T108 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 81676 T108 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 81681 T108 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 81728 T108 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 82326 T108 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 82347 T108 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 82351 T108 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 82408 T108 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 82419 T108 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 82429 T108 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 82433 T108 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 82434 T108 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 82434 T108 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 82438 T108 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 82438 T108 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 82439 T108 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 82439 T108 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1392562100495\collection1\, dataDir=.\org.apache.solr.cloud.BasicDistributedZkTest-1392562094463/jetty1\
   [junit4]   2> 82439 T108 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@53e3223e
   [junit4]   2> 82442 T108 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.BasicDistributedZkTest-1392562094463/jetty1\
   [junit4]   2> 82442 T108 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=.\org.apache.solr.cloud.BasicDistributedZkTest-1392562094463/jetty1\index/
   [junit4]   2> 82442 T108 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '.\org.apache.solr.cloud.BasicDistributedZkTest-1392562094463\jetty1\index' doesn't exist. Creating new index...
   [junit4]   2> 82443 T108 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.BasicDistributedZkTest-1392562094463/jetty1\index
   [junit4]   2> 82445 T108 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=697186585, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 82446 T108 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@3b5cfd86 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@501bf9ff),segFN=segments_1,generation=1}
   [junit4]   2> 82446 T108 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 82457 T108 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 82458 T108 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 82458 T108 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 82458 T108 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 82459 T108 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 82459 T108 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 82461 T108 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 82461 T108 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 82461 T108 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 82462 T108 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 82464 T108 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 82464 T108 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 82464 T108 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 82466 T108 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 82467 T108 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 82468 T108 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 82469 T108 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 82478 T108 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 82484 T108 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 82484 T108 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 82486 T108 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=48, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=2.0361328125, floorSegmentMB=0.515625, forceMergeDeletesPctAllowed=6.886370762905878, segmentsPerTier=22.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 82486 T108 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@3b5cfd86 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@501bf9ff),segFN=segments_1,generation=1}
   [junit4]   2> 82487 T108 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 82487 T108 oass.SolrIndexSearcher.<init> Opening Searcher@7debe95d[collection1] main
   [junit4]   2> 82502 T109 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7debe95d[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 82512 T108 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 82514 T60 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 82514 T60 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 82514 T112 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:50560 collection:collection1 shard:shard1
   [junit4]   2> 82519 T112 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 82548 T112 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 82557 T85 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 82560 T112 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 82560 T112 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C84 name=collection1 org.apache.solr.core.SolrCore@31274251 url=https://127.0.0.1:50560/collection1 node=127.0.0.1:50560_ C84_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:50560, core=collection1, node_name=127.0.0.1:50560_}
   [junit4]   2> 82562 T112 C84 P50560 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:50560/collection1/
   [junit4]   2> 82563 T112 C84 P50560 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 82564 T112 C84 P50560 oasc.SyncStrategy.syncToMe https://127.0.0.1:50560/collection1/ has no replicas
   [junit4]   2> 82564 T112 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:50560/collection1/ shard1
   [junit4]   2> 82564 T112 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 82573 T86 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 82608 T85 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 82611 T94 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 82612 T85 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 82612 T107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 82678 T86 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 82694 T85 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 82808 T94 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 82809 T85 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 82810 T107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 82826 T112 oasc.ZkController.register We are https://127.0.0.1:50560/collection1/ and leader is https://127.0.0.1:50560/collection1/
   [junit4]   2> 82826 T112 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:50560
   [junit4]   2> 82826 T112 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 82826 T112 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 82826 T112 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 82830 T85 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 82830 T85 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 82831 T85 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 82831 T112 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 82836 T86 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 82837 T86 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50560",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50560_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 82847 T85 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 82960 T85 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 82960 T94 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 82962 T107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 83116 T60 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 83117 T60 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 83131 T60 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 83140 T60 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:50569
   [junit4]   2> 83146 T60 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 83146 T60 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 83146 T60 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1392562104342
   [junit4]   2> 83146 T60 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1392562104342\'
   [junit4]   2> 83177 T60 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1392562104342\solr.xml
   [junit4]   2> 83244 T60 oasc.CoreContainer.<init> New CoreContainer 1342098341
   [junit4]   2> 83245 T60 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1392562104342\]
   [junit4]   2> 83246 T60 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 83246 T60 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 83246 T60 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 83246 T60 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 83246 T60 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 83246 T60 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 83248 T60 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 83248 T60 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 83248 T60 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 83265 T60 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 83265 T60 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 83265 T60 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 83267 T60 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50537/solr
   [junit4]   2> 83267 T60 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 83272 T60 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 83277 T123 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3c29cb33 name:ZooKeeperConnection Watcher:127.0.0.1:50537 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 83278 T60 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 83285 T60 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 83290 T125 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@65527cfc name:ZooKeeperConnection Watcher:127.0.0.1:50537/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 83290 T60 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 83303 T60 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 84352 T60 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50569_
   [junit4]   2> 84358 T60 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50569_
   [junit4]   2> 84376 T94 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 84376 T125 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 84376 T85 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 84376 T107 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 84401 T126 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 84401 T126 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 84405 T85 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 84405 T85 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 84405 T126 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 84405 T85 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 84409 T86 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 84411 T86 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50569",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50569_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2"

[...truncated too long message...]

 T7005 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 2023845 T7005 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDistributedSearch -Dtests.method=testDistribSearch -Dtests.seed=9FBBAC6CBCF702C4 -Dtests.slow=true -Dtests.locale=ja -Dtests.timezone=Africa/Lubumbashi -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   10.9s | TestDistributedSearch.testDistribSearch <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected mime type application/octet-stream but got text/html. <html>
   [junit4]    > <head>
   [junit4]    > <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
   [junit4]    > <title>Error 500 Server Error</title>
   [junit4]    > </head>
   [junit4]    > <body>
   [junit4]    > <h2>HTTP ERROR: 500</h2>
   [junit4]    > <p>Problem accessing /select. Reason:
   [junit4]    > <pre>    Server Error</pre></p>
   [junit4]    > <hr /><i><small>Powered by Jetty://</small></i>
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    > </body>
   [junit4]    > </html>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9FBBAC6CBCF702C4:1E5D2274CBA862F8]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:459)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
   [junit4]    > 	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:90)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.queryServer(BaseDistributedSearchTestCase.java:515)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:563)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:545)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:524)
   [junit4]    > 	at org.apache.solr.TestDistributedSearch.doTest(TestDistributedSearch.java:177)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:876)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 2023869 T7005 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Lucene45, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=ja, timezone=Africa/Lubumbashi
   [junit4]   2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.7.0_60-ea (64-bit)/cpus=2,threads=1,free=268527240,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [CursorPagingTest, SuggesterTSTTest, TestLRUCache, DocValuesMultiTest, BlockDirectoryTest, TestFaceting, BasicDistributedZkTest, DeleteShardTest, LoggingHandlerTest, AssignTest, DirectSolrConnectionTest, TestUtils, SchemaVersionSpecificBehaviorTest, TestArbitraryIndexDir, LukeRequestHandlerTest, TestCharFilters, TestDFRSimilarityFactory, TestMergePolicyConfig, CoreContainerCoreInitFailuresTest, TestRecovery, ExpressionTest, OverseerCollectionProcessorTest, ShardSplitTest, SpatialFilterTest, TestNumberUtils, IndexSchemaTest, TestRemoteStreaming, JSONWriterTest, MigrateRouteKeyTest, ReturnFieldsTest, TestDefaultSimilarityFactory, TestSuggestSpellingConverter, RAMDirectoryFactoryTest, BadComponentTest, ConvertedLegacyTest, LeaderElectionIntegrationTest, NoCacheHeaderTest, ZkControllerTest, DeleteReplicaTest, FileUtilsTest, DateMathParserTest, TestStandardQParsers, StatelessScriptUpdateProcessorFactoryTest, TestCSVLoader, ShardRoutingTest, ExternalFileFieldSortTest, PluginInfoTest, TestSolrJ, SoftAutoCommitTest, CachingDirectoryFactoryTest, TestCloudManagedSchemaAddField, PeerSyncTest, TestLFUCache, SliceStateUpdateTest, FullSolrCloudDistribCmdsTest, PrimUtilsTest, InfoHandlerTest, SuggesterTest, TestHighFrequencyDictionaryFactory, TermsComponentTest, AnalysisAfterCoreReloadTest, ResourceLoaderTest, AbstractAnalyticsFacetTest, TestIndexSearcher, HighlighterConfigTest, TestUniqueKeyFieldResource, TestRandomFaceting, SynonymTokenizerTest, LegacyHTMLStripCharFilterTest, TestClassNameShortening, DefaultValueUpdateProcessorTest, TestStressRecovery, IndexBasedSpellCheckerTest, TestInitQParser, TestPhraseSuggestions, SuggesterFSTTest, TestMaxScoreQueryParser, TestSimpleQParserPlugin, TestExtendedDismaxParser, JsonLoaderTest, TestRandomMergePolicy, CopyFieldTest, TestDynamicFieldCollectionResource, TestShardHandlerFactory, CSVRequestHandlerTest, HdfsDirectoryTest, DistribCursorPagingTest, TestSystemIdResolver, TestRandomDVFaceting, TestDynamicFieldResource, TestDistributedMissingSort, TestLMDirichletSimilarityFactory, DirectUpdateHandlerTest, TestDocumentBuilder, TestSolrXmlPersistence, SystemInfoHandlerTest, TestBinaryResponseWriter, TestStressLucene, TestRealTimeGet, BinaryUpdateRequestHandlerTest, DistributedSpellCheckComponentTest, TestCSVResponseWriter, TestCollationField, TestAddFieldRealTimeGet, ResponseLogComponentTest, TestRangeQuery, TestFieldResource, TestSearchPerf, HdfsCollectionsAPIDistributedZkTest, TestFiltering, DocValuesMissingTest, SearchHandlerTest, BlockCacheTest, OverseerTest, QueryElevationComponentTest, DistributedDebugComponentTest, TestConfig, TestSolrXMLSerializer, CurrencyFieldOpenExchangeTest, QueryEqualityTest, CurrencyFieldXmlFileTest, SOLR749Test, TestImplicitCoreProperties, NotRequiredUniqueKeyTest, TestPartialUpdateDeduplication, PreAnalyzedFieldTest, WordBreakSolrSpellCheckerTest, SimpleFacetsTest, TestCustomSort, TestSolrQueryParserDefaultOperatorResource, TestTrie, SpellCheckCollatorTest, CollectionsAPIDistributedZkTest, HdfsChaosMonkeySafeLeaderTest, SignatureUpdateProcessorFactoryTest, TestSolrDeletionPolicy1, AddBlockUpdateTest, DistanceFunctionTest, FieldMutatingUpdateProcessorTest, OpenCloseCoreStressTest, LeaderElectionTest, DistributedQueryElevationComponentTest, RegexBoostProcessorTest, CircularListTest, TestRecoveryHdfs, SyncSliceTest, SpellCheckComponentTest, CoreMergeIndexesAdminHandlerTest, BadCopyFieldTest, PreAnalyzedUpdateProcessorTest, UpdateParamsTest, TestNRTOpen, RequestHandlersTest, EnumFieldTest, TermVectorComponentDistributedTest, StressHdfsTest, ZkCLITest, AbstractAnalyticsStatsTest, SolrInfoMBeanTest, TestSchemaNameResource, TestLuceneMatchVersion, TestSchemaResource, TestAnalyzedSuggestions, RequiredFieldsTest, TimeZoneUtilsTest, TestDistributedGrouping, TestDefaultSearchFieldResource, TestPseudoReturnFields, TestManagedSchemaFieldResource, TestPHPSerializedResponseWriter, TestSerializedLuceneMatchVersion, TestHashPartitioner, TestSolrCoreProperties, TestSchemaVersionResource, ZkNodePropsTest, StandardRequestHandlerTest, TestCloudManagedSchema, DocValuesTest, MBeansHandlerTest, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, UnloadDistributedZkTest, BasicZkTest, RecoveryZkTest, TestReplicationHandler, ClusterStateUpdateTest, ZkSolrClientTest, TestZkChroot, ShardRoutingCustomTest, TestDistributedSearch]
   [junit4] Completed in 10.97s, 1 test, 1 error <<< FAILURES!

[...truncated 550 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:453: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:433: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:39: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-targets.xml:37: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build.xml:189: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\common-build.xml:491: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:1268: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:901: There were test failures: 372 suites, 1604 tests, 4 errors, 40 ignored (12 assumptions)

Total time: 79 minutes 52 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_60-ea-b04 -XX:+UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.8.0-fcs-b129) - Build # 3793 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/3793/
Java: 64bit/jdk1.8.0-fcs-b129 -XX:-UseCompressedOops -XX:+UseSerialGC

2 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
expected:<0> but was:<1>

Stack Trace:
java.lang.AssertionError: expected:<0> but was:<1>
	at __randomizedtesting.SeedInfo.seed([BAE03873D2EC8D72:3B06B66BA5B3ED4E]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:140)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)


REGRESSION:  org.apache.solr.cloud.ShardSplitTest.testDistribSearch

Error Message:
Wrong doc count on shard1_0 expected:<131> but was:<130>

Stack Trace:
java.lang.AssertionError: Wrong doc count on shard1_0 expected:<131> but was:<130>
	at __randomizedtesting.SeedInfo.seed([BAE03873D2EC8D72:3B06B66BA5B3ED4E]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.cloud.ShardSplitTest.checkDocCountsAndShardStates(ShardSplitTest.java:476)
	at org.apache.solr.cloud.ShardSplitTest.splitByUniqueKeyTest(ShardSplitTest.java:248)
	at org.apache.solr.cloud.ShardSplitTest.doTest(ShardSplitTest.java:112)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 10316 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> 907300 T4755 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 907302 T4755 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /hu_mhb/
   [junit4]   2> 907305 T4755 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\solrtest-ChaosMonkeySafeLeaderTest-1392730741719
   [junit4]   2> 907308 T4755 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 907308 T4756 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 907408 T4755 oasc.ZkTestServer.run start zk server on port:59391
   [junit4]   2> 907409 T4755 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 907415 T4762 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@89840f6 name:ZooKeeperConnection Watcher:127.0.0.1:59391 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 907416 T4755 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 907416 T4755 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 907421 T4757 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14445389be00000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 907424 T4755 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 907438 T4764 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ad16c34 name:ZooKeeperConnection Watcher:127.0.0.1:59391/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 907439 T4755 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 907439 T4755 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 907442 T4755 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 907445 T4755 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 907449 T4755 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 907452 T4755 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 907453 T4755 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 907459 T4755 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 907459 T4755 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 907464 T4755 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 907464 T4755 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 907468 T4755 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 907468 T4755 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 907473 T4755 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 907473 T4755 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 907477 T4755 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 907477 T4755 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 907482 T4755 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 907482 T4755 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 907487 T4755 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 907487 T4755 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 907492 T4755 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 907492 T4755 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 907496 T4755 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 907496 T4755 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 907500 T4755 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 907500 T4755 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 907508 T4757 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14445389be00001, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 907509 T4755 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 907525 T4766 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@22bde9bd name:ZooKeeperConnection Watcher:127.0.0.1:59391/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 907531 T4755 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 908075 T4755 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 908082 T4755 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 908111 T4755 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:59401
   [junit4]   2> 908114 T4755 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 908114 T4755 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 908114 T4755 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: .\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392730741947
   [junit4]   2> 908114 T4755 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '.\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392730741947\'
   [junit4]   2> 908145 T4755 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392730741947\solr.xml
   [junit4]   2> 908154 T4755 oasc.CoreContainer.<init> New CoreContainer 1230538479
   [junit4]   2> 908158 T4755 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=.\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392730741947\]
   [junit4]   2> 908158 T4755 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 908159 T4755 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 908159 T4755 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 908159 T4755 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 908160 T4755 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 908160 T4755 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 908160 T4755 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 908160 T4755 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 908161 T4755 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 908169 T4755 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 908170 T4755 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 908170 T4755 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 908170 T4755 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59391/solr
   [junit4]   2> 908171 T4755 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 908173 T4755 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 908181 T4777 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4ae2b4e9 name:ZooKeeperConnection Watcher:127.0.0.1:59391 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 908181 T4755 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 908186 T4755 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 908190 T4779 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3e28ffe0 name:ZooKeeperConnection Watcher:127.0.0.1:59391/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 908191 T4755 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 908193 T4755 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 908198 T4755 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 908206 T4755 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 908208 T4755 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59401_hu_mhb
   [junit4]   2> 908212 T4755 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59401_hu_mhb
   [junit4]   2> 908219 T4755 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 908224 T4755 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 908230 T4755 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:59401_hu_mhb
   [junit4]   2> 908230 T4755 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 908233 T4755 oasc.Overseer.start Overseer (id=91274001889886212-127.0.0.1:59401_hu_mhb-n_0000000000) starting
   [junit4]   2> 908240 T4755 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 908247 T4781 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 908249 T4755 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 908249 T4781 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 908252 T4755 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 908255 T4755 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 908258 T4780 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 908261 T4782 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 908261 T4782 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 908263 T4782 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 908263 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 908265 T4780 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 908266 T4780 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:59401/hu_mhb",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:59401_hu_mhb",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 908266 T4780 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 908267 T4780 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 908271 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 908273 T4779 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 909253 T4782 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 909253 T4782 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: .\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392730741947\collection1
   [junit4]   2> 909253 T4782 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 909255 T4782 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 909255 T4782 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 909257 T4782 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 909257 T4782 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '.\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392730741947\collection1\'
   [junit4]   2> 909259 T4782 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392730741947/collection1/lib/classes/' to classloader
   [junit4]   2> 909260 T4782 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392730741947/collection1/lib/README' to classloader
   [junit4]   2> 909289 T4782 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 909308 T4782 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 909312 T4782 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 909317 T4782 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 909442 T4782 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 909443 T4782 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 909446 T4782 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 909452 T4782 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 909484 T4782 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 909485 T4782 oasc.SolrCore.<init> [collection1] Opening new SolrCore at .\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392730741947\collection1\, dataDir=.\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392730741717/control/data\
   [junit4]   2> 909485 T4782 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@12c7e2ae
   [junit4]   2> 909486 T4782 oasc.CachingDirectoryFactory.get return new directory for C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392730741717\control\data
   [junit4]   2> 909487 T4782 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=.\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392730741717/control/data\index/
   [junit4]   2> 909488 T4782 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '.\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392730741717\control\data\index' doesn't exist. Creating new index...
   [junit4]   2> 909489 T4782 oasc.CachingDirectoryFactory.get return new directory for C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392730741717\control\data\index
   [junit4]   2> 909489 T4782 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=48, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 909496 T4782 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392730741717\control\data\index,segFN=segments_1,generation=1}
   [junit4]   2> 909496 T4782 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 909499 T4782 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 909499 T4782 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 909500 T4782 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 909500 T4782 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 909500 T4782 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 909500 T4782 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 909501 T4782 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 909501 T4782 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 909501 T4782 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 909502 T4782 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 909502 T4782 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 909502 T4782 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 909502 T4782 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 909503 T4782 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 909503 T4782 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 909503 T4782 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 909504 T4782 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 909508 T4782 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 909512 T4782 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 909513 T4782 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 909514 T4782 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=26, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.36795555119571854]
   [junit4]   2> 909516 T4782 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392730741717\control\data\index,segFN=segments_1,generation=1}
   [junit4]   2> 909516 T4782 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 909517 T4782 oass.SolrIndexSearcher.<init> Opening Searcher@25e3a141[collection1] main
   [junit4]   2> 909519 T4783 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@25e3a141[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 909524 T4782 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 909525 T4755 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 909525 T4755 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 909525 T4786 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:59401/hu_mhb collection:control_collection shard:shard1
   [junit4]   2> 909527 T4786 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 909534 T4755 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 909539 T4788 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10f86213 name:ZooKeeperConnection Watcher:127.0.0.1:59391/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 909539 T4755 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 909542 T4755 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 909548 T4786 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 909549 T4755 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 909557 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 909558 T4786 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 909558 T4786 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1004 name=collection1 org.apache.solr.core.SolrCore@1c9e5cb1 url=https://127.0.0.1:59401/hu_mhb/collection1 node=127.0.0.1:59401_hu_mhb C1004_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:59401/hu_mhb, core=collection1, node_name=127.0.0.1:59401_hu_mhb}
   [junit4]   2> 909558 T4786 C1004 P59401 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:59401/hu_mhb/collection1/
   [junit4]   2> 909558 T4786 C1004 P59401 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 909558 T4786 C1004 P59401 oasc.SyncStrategy.syncToMe https://127.0.0.1:59401/hu_mhb/collection1/ has no replicas
   [junit4]   2> 909558 T4786 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:59401/hu_mhb/collection1/ shard1
   [junit4]   2> 909558 T4786 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 909562 T4780 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 909569 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 909574 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 909580 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 909685 T4779 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 909686 T4788 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 909812 T4786 oasc.ZkController.register We are https://127.0.0.1:59401/hu_mhb/collection1/ and leader is https://127.0.0.1:59401/hu_mhb/collection1/
   [junit4]   2> 909812 T4786 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:59401/hu_mhb
   [junit4]   2> 909812 T4786 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 909812 T4786 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 909812 T4786 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 909840 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 909840 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 909840 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 910058 T4755 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 910059 T4755 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 910062 T4786 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 910069 T4755 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 910072 T4755 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:59413
   [junit4]   2> 910075 T4755 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 910075 T4755 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 910075 T4755 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: .\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392730743961
   [junit4]   2> 910075 T4755 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '.\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392730743961\'
   [junit4]   2> 910103 T4755 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392730743961\solr.xml
   [junit4]   2> 910115 T4755 oasc.CoreContainer.<init> New CoreContainer 386133579
   [junit4]   2> 910118 T4755 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=.\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392730743961\]
   [junit4]   2> 910118 T4755 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 910120 T4755 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 910120 T4755 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 910120 T4755 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 910120 T4755 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 910120 T4755 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 910120 T4755 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 910120 T4755 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 910121 T4755 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 910120 T4780 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 910122 T4780 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:59401/hu_mhb",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:59401_hu_mhb",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 910129 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 910131 T4755 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 910132 T4755 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 910132 T4755 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 910132 T4755 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59391/solr
   [junit4]   2> 910133 T4755 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 910134 T4755 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 910139 T4799 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@592271a5 name:ZooKeeperConnection Watcher:127.0.0.1:59391 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 910139 T4755 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 910143 T4755 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 910144 T4757 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14445389be00006, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 910148 T4801 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@34a949ab name:ZooKeeperConnection Watcher:127.0.0.1:59391/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 910148 T4755 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 910156 T4755 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 910235 T4779 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 910235 T4788 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 910235 T4801 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 911080 T4755 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59413_hu_mhb
   [junit4]   2> 911081 T4755 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59413_hu_mhb
   [junit4]   2> 911086 T4779 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 911086 T4788 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 911087 T4801 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 911097 T4802 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 911097 T4802 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 911099 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 911100 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 911100 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 911099 T4802 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 911101 T4780 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 911102 T4780 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:59413/hu_mhb",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:59413_hu_mhb",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 911103 T4780 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 911103 T4780 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 911108 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 911201 T4779 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 911201 T4788 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 911201 T4801 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 911991 T4802 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 911991 T4802 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: .\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392730743961\collection1
   [junit4]   2> 911991 T4802 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 911992 T4802 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 911992 T4802 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 911993 T4802 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 911993 T4802 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '.\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392730743961\collection1\'
   [junit4]   2> 911996 T4802 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392730743961/collection1/lib/classes/' to classloader
   [junit4]   2> 911997 T4802 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392730743961/collection1/lib/README' to classloader
   [junit4]   2> 912034 T4802 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 912041 T4802 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 912044 T4802 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 912048 T4802 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 912153 T4802 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 912155 T4802 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 912156 T4802 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 912160 T4802 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 912186 T4802 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 912186 T4802 oasc.SolrCore.<init> [collection1] Opening new SolrCore at .\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392730743961\collection1\, dataDir=.\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392730741717/jetty1\
   [junit4]   2> 912186 T4802 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@12c7e2ae
   [junit4]   2> 912187 T4802 oasc.CachingDirectoryFactory.get return new directory for C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392730741717\jetty1
   [junit4]   2> 912188 T4802 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=.\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392730741717/jetty1\index/
   [junit4]   2> 912189 T4802 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '.\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392730741717\jetty1\index' doesn't exist. Creating new index...
   [junit4]   2> 912190 T4802 oasc.CachingDirectoryFactory.get return new directory for C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392730741717\jetty1\index
   [junit4]   2> 912190 T4802 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=48, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 912195 T4802 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392730741717\jetty1\index,segFN=segments_1,generation=1}
   [junit4]   2> 912195 T4802 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 912197 T4802 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 912198 T4802 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 912198 T4802 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 912198 T4802 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 912198 T4802 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 912198 T4802 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 912200 T4802 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 912200 T4802 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 912200 T4802 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 912200 T4802 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 912200 T4802 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 912200 T4802 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 912200 T4802 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 912201 T4802 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 912201 T4802 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 912201 T4802 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 912202 T4802 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 912207 T4802 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 912209 T4802 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 912209 T4802 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 912210 T4802 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=26, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.36795555119571854]
   [junit4]   2> 912210 T4802 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392730741717\jetty1\index,segFN=segments_1,generation=1}
   [junit4]   2> 912210 T4802 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 912210 T4802 oass.SolrIndexSearcher.<init> Opening Searcher@d0f0e4a[collection1] main
   [junit4]   2> 912213 T4803 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@d0f0e4a[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 912214 T4802 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 912215 T4755 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 912216 T4755 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 912215 T4806 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:59413/hu_mhb collection:collection1 shard:shard2
   [junit4]   2> 912218 T4806 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 912262 T4806 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 912277 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 912277 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 912277 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 912279 T4806 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 912279 T4806 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1005 name=collection1 org.apache.solr.core.SolrCore@c3be5ac url=https://127.0.0.1:59413/hu_mhb/collection1 node=127.0.0.1:59413_hu_mhb C1005_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:59413/hu_mhb, core=collection1, node_name=127.0.0.1:59413_hu_mhb}
   [junit4]   2> 912279 T4806 C1005 P59413 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:59413/hu_mhb/collection1/
   [junit4]   2> 912279 T4806 C1005 P59413 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 912279 T4806 C1005 P59413 oasc.SyncStrategy.syncToMe https://127.0.0.1:59413/hu_mhb/collection1/ has no replicas
   [junit4]   2> 912279 T4806 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:59413/hu_mhb/collection1/ shard2
   [junit4]   2> 912279 T4806 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 912282 T4780 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 912290 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 912294 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 912300 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 912394 T4801 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 912395 T4779 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 912395 T4788 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 912432 T4806 oasc.ZkController.register We are https://127.0.0.1:59413/hu_mhb/collection1/ and leader is https://127.0.0.1:59413/hu_mhb/collection1/
   [junit4]   2> 912432 T4806 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:59413/hu_mhb
   [junit4]   2> 912432 T4806 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 912432 T4806 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 912432 T4806 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 912434 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 912434 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 912434 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 912436 T4806 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 912438 T4780 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 912439 T4780 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:59413/hu_mhb",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:59413_hu_mhb",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 912443 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 912537 T4779 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 912538 T4788 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 912538 T4801 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 912714 T4755 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 912714 T4755 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 912722 T4755 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 912722 T4755 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:59422
   [junit4]   2> 912726 T4755 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 912726 T4755 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 912728 T4755 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392730746632
   [junit4]   2> 912728 T4755 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392730746632\'
   [junit4]   2> 912752 T4755 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392730746632\solr.xml
   [junit4]   2> 912772 T4755 oasc.CoreContainer.<init> New CoreContainer 609116353
   [junit4]   2> 912772 T4755 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392730746632\]
   [junit4]   2> 912773 T4755 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 912773 T4755 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 912773 T4755 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 912774 T4755 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 912774 T4755 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 912774 T4755 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 912775 T4755 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 912775 T4755 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 912775 T4755 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 912783 T4755 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 912783 T4755 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 912783 T4755 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 912784 T4755 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59391/solr
   [junit4]   2> 912784 T4755 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 912786 T4755 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 912795 T4817 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@512025b9 name:ZooKeeperConnection Watcher:127.0.0.1:59391 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 912795 T4755 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 912799 T4755 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 912802 T4819 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3490f653 name:ZooKeeperConnection Watcher:127.0.0.1:59391/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 912803 T4755 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 912809 T4755 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 913704 T4755 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59422_hu_mhb
   [junit4]   2> 913706 T4755 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59422_hu_mhb
   [junit4]   2> 913710 T4779 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 913711 T4801 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 913711 T4819 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 913712 T4788 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 913719 T4820 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 913719 T4820 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 913720 T4820 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 913720 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 913720 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 913720 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 913722 T4780 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 913724 T4780 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:59422/hu_mhb",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:59422_hu_mhb",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 913724 T4780 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 913724 T4780 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 913728 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 913821 T4779 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 913821 T4801 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 913821 T4788 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 913821 T4819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 914611 T4820 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 914611 T4820 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392730746632\collection1
   [junit4]   2> 914611 T4820 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 914613 T4820 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 914613 T4820 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 914615 T4820 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 914615 T4820 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392730746632\collection1\'
   [junit4]   2> 914617 T4820 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392730746632/collection1/lib/classes/' to classloader
   [junit4]   2> 914618 T4820 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392730746632/collection1/lib/README' to classloader
   [junit4]   2> 914642 T4820 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 914652 T4820 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 914657 T4820 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 914662 T4820 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 914770 T4820 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 914771 T4820 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 914772 T4820 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 914775 T4820 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 914812 T4820 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 914812 T4820 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392730746632\collection1\, dataDir=.\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392730741717/jetty2\
   [junit4]   2> 914812 T4820 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@12c7e2ae
   [junit4]   2> 914813 T4820 oasc.CachingDirectoryFactory.get return new directory for C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392730741717\jetty2
   [junit4]   2> 914814 T4820 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=.\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392730741717/jetty2\index/
   [junit4]   2> 914815 T4820 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '.\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392730741717\jetty2\index' doesn't exist. Creating new index...
   [junit4]   2> 914815 T4820 oasc.CachingDirectoryFactory.get return new directory for C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392730741717\jetty2\index
   [junit4]   2> 914816 T4820 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=48, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 914821 T4820 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392730741717\jetty2\index,segFN=segments_1,generation=1}
   [junit4]   2> 914821 T4820 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 914823 T4820 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 914824 T4820 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 914824 T4820 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 914824 T4820 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 914824 T4820 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 914825 T4820 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 914825 T4820 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 914825 T4820 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 914825 T4820 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 914826 T4820 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 914826 T4820 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 914826 T4820 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 914826 T4820 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 914826 T4820 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 914826 T4820 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 914827 T4820 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 914827 T4820 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 914832 T4820 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 914834 T4820 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 914834 T4820 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 914835 T4820 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=26, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.36795555119571854]
   [junit4]   2> 914837 T4820 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392730741717\jetty2\index,segFN=segments_1,generation=1}
   [junit4]   2> 914837 T4820 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 914837 T4820 oass.SolrIndexSearcher.<init> Opening Searcher@50adbe38[collection1] main
   [junit4]   2> 914839 T4821 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@50adbe38[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 914840 T4820 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 914842 T4755 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 914843 T4755 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 914842 T4824 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:59422/hu_mhb collection:collection1 shard:shard3
   [junit4]   2> 914845 T4824 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 914855 T4824 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
   [junit4]   2> 914859 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 914859 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 914859 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 914860 T4824 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 914860 T4824 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1006 name=collection1 org.apache.solr.core.SolrCore@28fba5bb url=https://127.0.0.1:59422/hu_mhb/collection1 node=127.0.0.1:59422_hu_mhb C1006_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:59422/hu_mhb, core=collection1, node_name=127.0.0.1:59422_hu_mhb}
   [junit4]   2> 914860 T4824 C1006 P59422 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:59422/hu_mhb/collection1/
   [junit4]   2> 914860 T4824 C1006 P59422 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 914860 T4824 C1006 P59422 oasc.SyncStrategy.syncToMe https://127.0.0.1:59422/hu_mhb/collection1/ has no replicas
   [junit4]   2> 914860 T4824 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:59422/hu_mhb/collection1/ shard3
   [junit4]   2> 914860 T4824 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 914865 T4780 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 914874 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 914879 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 914885 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 914988 T4779 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 914989 T4819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 914989 T4788 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 914989 T4801 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 915021 T4824 oasc.ZkController.register We are https://127.0.0.1:59422/hu_mhb/collection1/ and leader is https://127.0.0.1:59422/hu_mhb/collection1/
   [junit4]   2> 915021 T4824 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:59422/hu_mhb
   [junit4]   2> 915021 T4824 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 915021 T4824 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 915021 T4824 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 915023 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 915023 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 915023 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 915025 T4824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 915026 T4780 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 915027 T4780 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:59422/hu_mhb",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:59422_hu_mhb",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 915030 T4779 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 915124 T4779 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 915124 T4801 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 915125 T4788 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 915124 T4819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 915288 T4755 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 915288 T4755 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 915301 T4755 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 915303 T4755 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:59431
   [junit4]   2> 915309 T4755 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 915310 T4755 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 915310 T4755 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392730749258
   [junit4]   2> 915311 T4755 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392730749258\'
   [junit4]   2> 915337 T4755 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392730749258\solr.xml
   [junit4]   2> 915348 T4755 oasc.CoreContainer.<init> New CoreContainer 118027510
   [junit4]   2> 915348 T4755 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392730749258\]
   [junit4]   2> 915348 T4755 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 915349 T4755 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 915349 T4755 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 915349 T4755 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 915350 T4755 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 915350 T4755 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 915350 T4755 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 915350 T4755 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 915351 T4755 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 915359 T4755 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 915359 T4755 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 915360 T4755 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 915360 T4755 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59391/solr
   [junit4]   2> 915360 T4755 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 915361 T4755 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 915369 T4835 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6f609263 name:ZooKeeperConnection Watcher:127.0.0.1:59391 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 915369 T4755 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 915374 T4755 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 915376 T4837 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2339f81a name:ZooKeeperConnection Watcher:127.0.0.1:59391/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 915377 T4755 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 915384 T4755 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 916282 T4755 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59431_hu_mhb
   [junit4]   2> 916285 T4755 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59431_hu_mhb
   [junit4]   2> 916289 T4779 oascc.ZkStateReader$3.p

[...truncated too long message...]

Dtests.slow=true -Dtests.locale=el_CY -Dtests.timezone=America/Toronto -Dtests.file.encoding=Cp1252
   [junit4] FAILURE 37.3s | ShardSplitTest.testDistribSearch <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Wrong doc count on shard1_0 expected:<131> but was:<130>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([BAE03873D2EC8D72:3B06B66BA5B3ED4E]:0)
   [junit4]    > 	at org.apache.solr.cloud.ShardSplitTest.checkDocCountsAndShardStates(ShardSplitTest.java:476)
   [junit4]    > 	at org.apache.solr.cloud.ShardSplitTest.splitByUniqueKeyTest(ShardSplitTest.java:248)
   [junit4]    > 	at org.apache.solr.cloud.ShardSplitTest.doTest(ShardSplitTest.java:112)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 2756189 T11419 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 37306 T11418 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting, sim=DefaultSimilarity, locale=el_CY, timezone=America/Toronto
   [junit4]   2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.8.0 (64-bit)/cpus=2,threads=1,free=135310392,total=389332992
   [junit4]   2> NOTE: All tests run in this JVM: [TestHighFrequencyDictionaryFactory, TestValueSourceCache, TriLevelCompositeIdRoutingTest, OpenExchangeRatesOrgProviderTest, SoftAutoCommitTest, TermVectorComponentTest, TestAtomicUpdateErrorCases, CopyFieldTest, SuggesterTSTTest, NoFacetTest, TestCSVLoader, TestSort, HardAutoCommitTest, BasicFunctionalityTest, SolrPluginUtilsTest, TestCoreDiscovery, HdfsChaosMonkeySafeLeaderTest, TestFuzzyAnalyzedSuggestions, TestQuerySenderListener, FileUtilsTest, SynonymTokenizerTest, TestAnalyzeInfixSuggestions, TestFunctionQuery, TestNonDefinedSimilarityFactory, SolrXmlInZkTest, TestPartialUpdateDeduplication, UnloadDistributedZkTest, TestSchemaVersionResource, PeerSyncTest, LeaderElectionIntegrationTest, DateMathParserTest, RemoteQueryErrorTest, TestSolrIndexConfig, TestDistributedSearch, PingRequestHandlerTest, TestPseudoReturnFields, TestDynamicFieldCollectionResource, TestLMJelinekMercerSimilarityFactory, TestBlendedInfixSuggestions, CollectionsAPIDistributedZkTest, TestOverriddenPrefixQueryForCustomFieldType, TestStressLucene, DocumentBuilderTest, TestGroupingSearch, TestFieldTypeCollectionResource, TestElisionMultitermQuery, TestAddFieldRealTimeGet, TestSolrXml, TestCodecSupport, TestUtils, UpdateParamsTest, TestDistributedGrouping, TestMaxScoreQueryParser, AliasIntegrationTest, SOLR749Test, TestSolrXMLSerializer, SliceStateTest, StressHdfsTest, EnumFieldTest, TestIndexSearcher, ZkCLITest, TestFileDictionaryLookup, TestReplicationHandler, HdfsUnloadDistributedZkTest, TestExtendedDismaxParser, RAMDirectoryFactoryTest, DocValuesMultiTest, QueryFacetTest, TestSchemaNameResource, DistributedDebugComponentTest, DirectSolrSpellCheckerTest, TestReload, TestCharFilters, SystemInfoHandlerTest, TestSolrCoreProperties, SolrCoreTest, TestCustomSort, TestDynamicFieldResource, SuggesterWFSTTest, SimpleFacetsTest, TestSerializedLuceneMatchVersion, TestUpdate, CSVRequestHandlerTest, TestCSVResponseWriter, TestLMDirichletSimilarityFactory, TestCloudManagedSchemaAddField, DistributedSuggestComponentTest, TestFastLRUCache, TestSchemaSimilarityResource, TestManagedSchemaFieldResource, TestBinaryField, FileBasedSpellCheckerTest, TermVectorComponentDistributedTest, OutputWriterTest, TestSolrXmlPersistor, TestMergePolicyConfig, UniqFieldsUpdateProcessorFactoryTest, AlternateDirectoryTest, ZkControllerTest, TestPostingsSolrHighlighter, TestNonNRTOpen, TestSuggestSpellingConverter, BJQParserTest, LoggingHandlerTest, TestFreeTextSuggestions, TestSolrXmlPersistence, SortByFunctionTest, DOMUtilTest, AddBlockUpdateTest, TestRandomFaceting, CoreAdminCreateDiscoverTest, StatsComponentTest, TestStressUserVersions, PreAnalyzedFieldTest, TestZkChroot, TestSolrDeletionPolicy2, ShowFileRequestHandlerTest, TestCoreContainer, TestSimpleQParserPlugin, MultiTermTest, QueryParsingTest, MinimalSchemaTest, AnalysisAfterCoreReloadTest, SolrIndexConfigTest, DistributedSpellCheckComponentTest, TestReversedWildcardFilterFactory, AbstractAnalyticsFacetTest, TestSolrQueryParser, ChaosMonkeySafeLeaderTest, SpellCheckCollatorTest, AnalysisErrorHandlingTest, TestPerFieldSimilarity, DocValuesTest, BasicZkTest, HdfsSyncSliceTest, DeleteShardTest, LegacyHTMLStripCharFilterTest, IndexSchemaTest, SolrIndexSplitterTest, TestFaceting, OverseerTest, FieldFacetExtrasTest, TestSolrQueryParserDefaultOperatorResource, TestPhraseSuggestions, TestRecovery, NoCacheHeaderTest, SchemaVersionSpecificBehaviorTest, BadIndexSchemaTest, TestStressVersions, ExpressionTest, CachingDirectoryFactoryTest, PolyFieldTest, SpatialFilterTest, DistribCursorPagingTest, SolrInfoMBeanTest, HighlighterTest, TestJmxIntegration, TestLuceneMatchVersion, TestSearchPerf, WordBreakSolrSpellCheckerTest, TestNoOpRegenerator, TestFastWriter, TestRecoveryHdfs, TestSystemIdResolver, RegexBoostProcessorTest, FastVectorHighlighterTest, TestFastOutputStream, SignatureUpdateProcessorFactoryTest, ScriptEngineTest, ZkNodePropsTest, URLClassifyProcessorTest, ClusterStateTest, SuggestComponentTest, CustomCollectionTest, DocumentAnalysisRequestHandlerTest, RecoveryZkTest, BasicDistributedZkTest, BasicDistributedZk2Test, SyncSliceTest, OpenCloseCoreStressTest, ShardRoutingTest, FullSolrCloudDistribCmdsTest, ClusterStateUpdateTest, LeaderElectionTest, ZkSolrClientTest, TestRandomDVFaceting, ShardRoutingCustomTest, SliceStateUpdateTest, AutoCommitTest, TestHashPartitioner, TestRealTimeGet, TestStressReorder, TestJoin, TestMultiCoreConfBootstrap, DistributedTermsComponentTest, TestRangeQuery, TestSolr4Spatial, SpellCheckComponentTest, SolrCmdDistributorTest, QueryElevationComponentTest, ConvertedLegacyTest, TestFiltering, TestLazyCores, DirectUpdateHandlerTest, TestBadConfig, OverseerCollectionProcessorTest, CurrencyFieldOpenExchangeTest, DistributedQueryElevationComponentTest, CurrencyFieldXmlFileTest, SimplePostToolTest, SuggesterFSTTest, CoreAdminHandlerTest, SolrRequestParserTest, TestFoldingMultitermQuery, SuggesterTest, TestTrie, SolrCoreCheckLockOnStartupTest, FieldMutatingUpdateProcessorTest, TestWordDelimiterFilterFactory, QueryEqualityTest, DirectUpdateHandlerOptimizeTest, DefaultValueUpdateProcessorTest, StatelessScriptUpdateProcessorFactoryTest, TestRemoteStreaming, DistanceFunctionTest, TestSolrDeletionPolicy1, XsltUpdateRequestHandlerTest, DebugComponentTest, CacheHeaderTest, IndexBasedSpellCheckerTest, TestSurroundQueryParser, LukeRequestHandlerTest, DisMaxRequestHandlerTest, TestQueryUtils, StandardRequestHandlerTest, TestWriterPerf, TestQueryTypes, PrimitiveFieldTypeTest, TestOmitPositions, TermsComponentTest, XmlUpdateRequestHandlerTest, RequestHandlersTest, PathHierarchyTokenizerFactoryTest, TestIndexingPerformance, MoreLikeThisHandlerTest, RequiredFieldsTest, FieldAnalysisRequestHandlerTest, TestArbitraryIndexDir, IndexSchemaRuntimeFieldTest, TestCollationField, ReturnFieldsTest, UpdateRequestProcessorFactoryTest, JSONWriterTest, MBeansHandlerTest, TestAnalyzedSuggestions, JsonLoaderTest, BinaryUpdateRequestHandlerTest, TestPHPSerializedResponseWriter, TestComponentsName, SearchHandlerTest, CoreContainerCoreInitFailuresTest, TestBinaryResponseWriter, TestLFUCache, HighlighterConfigTest, TestQuerySenderNoQuery, ResponseLogComponentTest, BadComponentTest, TestStressRecovery, SampleTest, TestDocSet, NumericFieldsTest, TestConfig, ExternalFileFieldSortTest, DirectSolrConnectionTest, NotRequiredUniqueKeyTest, SpellPossibilityIteratorTest, TestXIncludeConfig, EchoParamsTest, TestSweetSpotSimilarityFactory, TestDFRSimilarityFactory, TestNumberUtils, TestIBSimilarityFactory, TestBM25SimilarityFactory, TestDefaultSimilarityFactory, TestJmxMonitoredMap, TimeZoneUtilsTest, ResourceLoaderTest, PluginInfoTest, ChaosMonkeyNothingIsSafeTest, PrimUtilsTest, DateFieldTest, SpellingQueryConverterTest, TestSolrJ, TestLRUCache, TestDocumentBuilder, UUIDFieldTest, CircularListTest, TestRTGBase, CursorPagingTest, SolrTestCaseJ4Test, TestCursorMarkWithoutUniqueKey, TestDistributedMissingSort, TestHighlightDedupGrouping, AbstractAnalyticsStatsTest, RangeFacetTest, FunctionTest, AssignTest, ConnectionManagerTest, DeleteReplicaTest, MigrateRouteKeyTest, OverseerRolesTest, ShardSplitTest]
   [junit4] Completed in 37.50s, 1 test, 1 failure <<< FAILURES!

[...truncated 145 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:453: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:433: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:39: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-targets.xml:37: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build.xml:189: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\common-build.xml:491: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:1268: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:901: There were test failures: 372 suites, 1604 tests, 2 failures, 40 ignored (12 assumptions)

Total time: 73 minutes 52 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0-fcs-b129 -XX:-UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Windows (32bit/jdk1.7.0_51) - Build # 3792 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/3792/
Java: 32bit/jdk1.7.0_51 -client -XX:+UseG1GC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.OverseerTest.testOverseerFailure

Error Message:
KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1

Stack Trace:
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
	at __randomizedtesting.SeedInfo.seed([CB90248002EF6C16:CF98AB73104A8337]:0)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:429)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:426)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:383)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:370)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:156)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:156)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:153)
	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:584)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 10447 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerTest
   [junit4]   2> 2090707 T5814 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 2090707 T5814 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> Creating dataDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\solrtest-OverseerTest-1392707057342
   [junit4]   2> 2090709 T5814 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 2090713 T5814 oas.SolrTestCaseJ4.setUp ###Starting testPlaceholders
   [junit4]   2> 2090714 T5814 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 2090716 T5815 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2090813 T5814 oasc.ZkTestServer.run start zk server on port:55304
   [junit4]   2> 2090815 T5814 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2090833 T5821 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f1f12b name:ZooKeeperConnection Watcher:127.0.0.1:55304/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2090835 T5814 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2090840 T5814 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2090847 T5823 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ecb19f name:ZooKeeperConnection Watcher:127.0.0.1:55304 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2090847 T5814 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2090851 T5816 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14443cf36d00001, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 2090852 T5814 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2090857 T5825 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f2e17 name:ZooKeeperConnection Watcher:127.0.0.1:55304 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2090857 T5814 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2090857 T5814 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2090863 T5814 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2090863 T5816 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14443cf36d00002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 2090867 T5814 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2090871 T5814 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2090874 T5814 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2090878 T5814 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2090881 T5827 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2cb438 name:ZooKeeperConnection Watcher:127.0.0.1:55304/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2090882 T5814 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2090884 T5814 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2090888 T5814 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 2090892 T5821 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 2090892 T5827 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 2090893 T5814 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2090897 T5829 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1dc5d4d name:ZooKeeperConnection Watcher:127.0.0.1:55304/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2090897 T5814 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2090899 T5814 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2090902 T5814 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2090908 T5814 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:55304_solr
   [junit4]   2> 2090908 T5814 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2090912 T5814 oasc.Overseer.start Overseer (id=91272449711144964-127.0.0.1:55304_solr-n_0000000000) starting
   [junit4]   2> 2090916 T5814 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2090921 T5814 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2090925 T5814 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2090930 T5831 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2090931 T5831 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 2090932 T5830 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2090936 T5830 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2090937 T5830 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=12 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"12",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 2090937 T5830 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3, shard4, shard5, shard6, shard7, shard8, shard9, shard10, shard11, shard12]
   [junit4]   2> 2090937 T5830 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard11
   [junit4]   2> 2090940 T5829 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2090942 T5827 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2090942 T5821 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2091413 T5814 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard11/election
   [junit4]   2> 2091437 T5814 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard11
   [junit4]   2> 2091448 T5829 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2091450 T5814 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2091452 T5830 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2091453 T5816 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14443cf36d00004, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 2091457 T5821 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 2091457 T5827 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 2091458 T5816 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14443cf36d00003, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 2091463 T5816 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14443cf36d00000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 2091464 T5814 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:55304 55304
   [junit4]   2> 2092893 T5830 oasc.Overseer$ClusterStateUpdater.run WARN Solr cannot talk to ZK, exiting Overseer main queue loop org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterstate.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1151)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:271)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:271)
   [junit4]   2> 	at org.apache.solr.common.cloud.ClusterState.load(ClusterState.java:239)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:408)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:226)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:180)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 2093633 T5814 oas.SolrTestCaseJ4.tearDown ###Ending testPlaceholders
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2093645 T5814 oas.SolrTestCaseJ4.setUp ###Starting testDoubleAssignment
   [junit4]   2> 2093645 T5814 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 2093648 T5832 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2093745 T5814 oasc.ZkTestServer.run start zk server on port:55322
   [junit4]   2> 2093748 T5814 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2093761 T5838 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7cac52 name:ZooKeeperConnection Watcher:127.0.0.1:55322/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2093761 T5814 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2093766 T5814 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2093771 T5840 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9b56f9 name:ZooKeeperConnection Watcher:127.0.0.1:55322 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2093772 T5814 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2093839 T5833 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14443cf42430001, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 2093842 T5814 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2093845 T5842 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1454fbd name:ZooKeeperConnection Watcher:127.0.0.1:55322 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2093845 T5814 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2093845 T5814 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2093849 T5814 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2093849 T5833 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14443cf42430002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 2093854 T5814 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2093858 T5814 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2093861 T5814 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2093865 T5814 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2093868 T5844 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@102bc3c name:ZooKeeperConnection Watcher:127.0.0.1:55322/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2093868 T5814 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2093870 T5814 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2093874 T5814 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 2093878 T5838 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 2093879 T5844 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 2093880 T5814 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2093883 T5846 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ea8412 name:ZooKeeperConnection Watcher:127.0.0.1:55322/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2093884 T5814 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2093885 T5814 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2093889 T5814 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2093895 T5814 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:55322_solr
   [junit4]   2> 2093895 T5814 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2093899 T5814 oasc.Overseer.start Overseer (id=91272449903230980-127.0.0.1:55322_solr-n_0000000000) starting
   [junit4]   2> 2093902 T5814 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2093907 T5814 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2093913 T5814 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2093918 T5848 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2093919 T5848 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 2093920 T5847 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2093922 T5847 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2093924 T5847 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 2093924 T5847 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
   [junit4]   2> 2093924 T5847 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2093929 T5846 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2093931 T5844 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2093931 T5838 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2094400 T5814 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 2094416 T5814 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2094426 T5846 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2094427 T5814 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2094429 T5847 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2094432 T5844 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 2094432 T5838 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 2094434 T5833 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14443cf42430003, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 2094436 T5814 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2094437 T5846 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2094439 T5850 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10980f1 name:ZooKeeperConnection Watcher:127.0.0.1:55322/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2094440 T5814 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2094442 T5814 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2094445 T5814 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 2094451 T5850 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 2094452 T5838 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 2094453 T5846 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2094454 T5814 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 2094456 T5847 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 2094456 T5847 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 2094463 T5846 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2094464 T5814 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2094471 T5846 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2094481 T5846 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2094581 T5850 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2094581 T5838 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2094583 T5814 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2094586 T5833 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14443cf42430004, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 2094589 T5850 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 2094589 T5838 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 2094594 T5814 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:55322 55322
   [junit4]   2> 2095960 T5847 oasc.Overseer$ClusterStateUpdater.run WARN Solr cannot talk to ZK, exiting Overseer main queue loop org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue-work/qn-0000000002
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1151)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:271)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:271)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.remove(DistributedQueue.java:169)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.poll(DistributedQueue.java:445)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:201)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 2096623 T5814 oas.SolrTestCaseJ4.tearDown ###Ending testDoubleAssignment
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2096634 T5814 oas.SolrTestCaseJ4.setUp ###Starting testOverseerFailure
   [junit4]   2> 2096635 T5814 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 2096636 T5851 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2096723 T5814 oasc.ZkTestServer.run start zk server on port:55343
   [junit4]   2> 2096724 T5814 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2096732 T5857 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6264d name:ZooKeeperConnection Watcher:127.0.0.1:55343/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2096733 T5814 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2096734 T5814 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2096738 T5859 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f63cae name:ZooKeeperConnection Watcher:127.0.0.1:55343 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2096739 T5814 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2096784 T5852 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14443cf4df00001, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 2096786 T5814 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2096789 T5861 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fdd58d name:ZooKeeperConnection Watcher:127.0.0.1:55343 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2096790 T5814 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2096790 T5814 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2096794 T5814 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2096799 T5814 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2096804 T5814 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2096807 T5814 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2096814 T5814 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2096817 T5863 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@da35c7 name:ZooKeeperConnection Watcher:127.0.0.1:55343/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2096818 T5814 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2096820 T5814 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2096824 T5814 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 2096830 T5857 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 2096831 T5863 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 2096832 T5814 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2096835 T5865 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c72e51 name:ZooKeeperConnection Watcher:127.0.0.1:55343/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2096836 T5814 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2096837 T5814 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2096842 T5814 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2096850 T5814 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:55343_solr
   [junit4]   2> 2096851 T5814 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2096855 T5814 oasc.Overseer.start Overseer (id=91272450099118084-127.0.0.1:55343_solr-n_0000000000) starting
   [junit4]   2> 2096859 T5814 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2096865 T5814 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2096874 T5814 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2096879 T5867 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2096881 T5867 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 2096881 T5866 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2097747 T5865 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2097750 T5866 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2097751 T5866 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 2097751 T5866 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
   [junit4]   2> 2097751 T5866 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2097756 T5865 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2097757 T5857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2097758 T5863 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2098180 T5814 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 2098193 T5814 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2098201 T5865 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2098202 T5814 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2098204 T5866 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2098207 T5865 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2098207 T5814 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 2098209 T5866 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 2098210 T5866 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 2098214 T5865 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2098303 T5863 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2098303 T5857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2099219 T5814 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 2099233 T5814 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2099236 T5869 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@64565d name:ZooKeeperConnection Watcher:127.0.0.1:55343/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2099237 T5814 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2099245 T5814 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:55343_solr
   [junit4]   2> 2099245 T5814 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2099250 T5814 oasc.Overseer.start Overseer (id=91272450099118085-127.0.0.1:55343_solr-n_0000000001) starting
   [junit4]   2> 2099258 T5871 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2099260 T5871 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 2099261 T5870 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2099264 T5870 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2099266 T5870 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 2099266 T5870 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 2099270 T5869 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2099271 T5857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2099272 T5863 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2099277 T5814 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 2099279 T5870 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2099287 T5814 oasc.LeaderElector.checkIfIamLeader WARN Failed setting watch org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election/91272450099118083-node1_core1-n_0000000002
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1151)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:271)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:271)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:153)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:584)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 2099291 T5814 oasc.LeaderElector.checkIfIamLeader WARN Failed setting watch org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election/91272450099118083-node1_core1-n_0000000001
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1151)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:271)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:271)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:156)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:153)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:584)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 2099293 T5870 oasc.SolrException.log ERROR Problem cleaning up collection in zk:collection1:org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:125)
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 		at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:174)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:171)
   [junit4]   2> 		at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:171)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:633)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:628)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:628)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:628)
   [junit4]   2> 		at org.apache.solr.cloud.Overseer$ClusterStateUpdater.removeCore(Overseer.java:879)
   [junit4]   2> 		at org.apache.solr.cloud.Overseer$ClusterStateUpdater.processMessage(Overseer.java:226)
   [junit4]   2> 		at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:187)
   [junit4]   2> 		at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	
   [junit4]   2> 2099293 T5863 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2099297 T5814 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2099299 T5869 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2099305 T5869 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2099308 T5857 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 2099310 T5863 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2099310 T5870 oasc.Overseer$ClusterStateUpdater.setShardLeader ERROR Could not mark shard leader for non existing collection:collection1
   [junit4]   2> 2099313 T5852 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14443cf4df00005, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 2099315 T5852 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14443cf4df00000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 2099318 T5814 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:55343 55343
   [junit4]   2> 2100606 T5863 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:206)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:203)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:203)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:414)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:383)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:370)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 2100608 T5870 oasc.Overseer$ClusterStateUpdater.run WARN Solr cannot talk to ZK, exiting Overseer main queue loop org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue-work/qn-
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:240)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:237)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:237)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.createData(DistributedQueue.java:284)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.offer(DistributedQueue.java:271)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:188)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 2101267 T5863 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 2101269 T5814 oas.SolrTestCaseJ4.tearDown ###Ending testOverseerFailure
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=OverseerTest -Dtests.method=testOverseerFailure -Dtests.seed=CB90248002EF6C16 -Dtests.slow=true -Dtests.locale=et_EE -Dtests.timezone=Etc/GMT+8 -Dtests.file.encoding=UTF-8
   [junit4] ERROR   4.66s | OverseerTest.testOverseerFailure <<<
   [junit4]    > Throwable #1: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([CB90248002EF6C16:CF98AB73104A8337]:0)
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]    > 	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:429)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:426)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:383)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:370)
   [junit4]    > 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:156)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:156)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
   [junit4]    > 	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:153)
   [junit4]    > 	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:584)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 2101290 T5814 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignment
   [junit4]   2> 2101290 T5814 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 2101292 T5872 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2101379 T5814 oasc.ZkTestServer.run start zk server on port:55364
   [junit4]   2> 2101381 T5814 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2101387 T5878 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@beab2b name:ZooKeeperConnection Watcher:127.0.0.1:55364 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2101387 T5814 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2101428 T5873 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14443cf601e0000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 2101430 T5814 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2101433 T5880 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1bfc643 name:ZooKeeperConnection Watcher:127.0.0.1:55364 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2101433 T5814 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2101433 T5814 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2101438 T5873 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14443cf601e0001, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 2101440 T5814 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2101444 T5882 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1cbc037 name:ZooKeeperConnection Watcher:127.0.0.1:55364/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2101444 T5814 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2101444 T5814 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2101450 T5814 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2101454 T5884 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b41b9c name:ZooKeeperConnection Watcher:127.0.0.1:55364/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2101455 T5814 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2101457 T5814 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2101461 T5814 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2101469 T5814 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:55364_solr
   [junit4]   2> 2101469 T5814 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2101475 T5814 oasc.Overseer.start Overseer (id=91272450404122627-127.0.0.1:55364_solr-n_0000000000) starting
   [junit4]   2> 2101479 T5814 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2101486 T5814 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2101493 T5814 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2101498 T5886 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2101498 T5814 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2101500 T5885 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2101500 T5886 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 2101504 T5814 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2101507 T5814 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2101513 T5814 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2101516 T5888 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d5f2a name:ZooKeeperConnection Watcher:127.0.0.1:55364/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2101517 T5814 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2101518 T5814 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2101522 T5814 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1
   [junit4]   2> 2101527 T5888 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 2101528 T5882 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 2101533 T5884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2101536 T5885 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2101537 T5885 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 2101537 T5885 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 2101537 T5885 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 2101542 T5884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2101543 T5882 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2101544 T5888 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2101966 T5814 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 2101983 T5814 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 2101994 T5884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2101997 T5885 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2101999 T5884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2102006 T5885 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core2",
   [junit4]   2> 	  "core_node_name":"node2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 2102006 T5885 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 2102007 T5885 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2102013 T5884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2102103 T5882 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2102104 T5888 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2102432 T5814 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 2102447 T5814 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2102458 T5884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2102459 T5884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2102459 T5884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2102463 T5884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2102464 T5885 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2102473 T5885 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core3",
   [junit4]   2> 	  "core_node_name":"node3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 2102473 T5885 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 2102474 T5885 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 2102479 T5884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2102570 T5888 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2102571 T5882 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2102895 T5814 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 2102911 T5814 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 2102920 T5884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2102921 T5884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2102921 T5884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2102925 T5884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2102926 T5885 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2102933 T5885 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core4",
   [junit4]   2> 	  "core_node_name":"node4",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 2102933 T5885 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 2102933 T5885 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 2102939 T5884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2103030 T5882 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2103030 T5888 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2103357 T5814 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 2103382 T5884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2103382 T5884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2103383 T5884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2103388 T5885 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2103390 T5885 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core5",
   [junit4]   2> 	  "core_node_name":"node5",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 2103390 T5885 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 2103390 T5885 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2103399 T5884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2103491 T5882 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2103491 T5888 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2103815 T5814 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 2103842 T5884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2103843 T5884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2103843 T5884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2103848 T5885 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2103849 T5885 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core6",
   [junit4]   2> 	  "core_node_name":"node6",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 2103849 T5885 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 2103849 T5885 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 2103859 T5884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2103950 T5888 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2103951 T5882 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2104249 T5814 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 2104258 T5873 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14443cf601e0002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 2104261 T5888 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 2104265 T5814 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:55364 55364
   [junit4]   2> 2105429 T5888 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 2106020 T5888 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 2106023 T5814 oas.SolrTestCaseJ4.tearDown ###Ending testShardAssignment
   [junit4]   2> 2106024 T5888 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2106036 T5814 oas.SolrTestCaseJ4.setUp ###Starting testShardLeaderChange
   [junit4]   2> 2106036 T5814 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 2106039 T5889 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2106117 T5814 oasc.ZkTestServer.run start zk server on port:55382
   [junit4]   2> 2106120 T5814 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2106130 T5895 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f86a9c name:ZooKeeperConnection Watcher:127.0.0.1:55382/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2106131 T5814 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2106134 T5814 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2106140 T5897 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@198bb31 name:ZooKeeperConnection Watcher:127.0.0.1:55382 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2106140 T5814 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2106212 T5814 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2106216 T5899 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@86b6f1 name:ZooKeeperConnection Watcher:127.0.0.1:55382 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2106217 T5814 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2106218 T5814 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2106226 T5814 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2106226 T5890 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14443cf72aa0002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at o

[...truncated too long message...]

lectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 2123762 T6239 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2123763 T6239 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue.
   [junit4]   2> 2123764 T6239 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "shard":"s1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 2123766 T6232 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 2123770 T6239 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "shard":"s1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core2",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 2123772 T6232 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 2123776 T6239 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2123779 T6239 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2123780 T6239 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "shard":"s1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core3",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 2123783 T6238 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2123785 T6232 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 2123859 T6227 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14443cfb6d90003, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 2123863 T6227 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14443cfb6d90000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 2123865 T5814 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:55724 55724
   [junit4]   2> 2125989 T5814 oas.SolrTestCaseJ4.tearDown ###Ending testReplay
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2125993 T5814 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 2125993 T5814 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 2128820 T5814 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Lucene42, sim=DefaultSimilarity, locale=et_EE, timezone=Etc/GMT+8
   [junit4]   2> NOTE: Windows 7 6.1 x86/Oracle Corporation 1.7.0_51 (32-bit)/cpus=2,threads=1,free=142958256,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [TestRemoteStreaming, SolrCoreTest, ZkCLITest, TestSolrDeletionPolicy2, SchemaVersionSpecificBehaviorTest, XsltUpdateRequestHandlerTest, TestPostingsSolrHighlighter, TestXIncludeConfig, SolrCmdDistributorTest, StatsComponentTest, OpenExchangeRatesOrgProviderTest, TestJoin, TermsComponentTest, CoreMergeIndexesAdminHandlerTest, TestSolrXml, TestManagedSchema, SystemInfoHandlerTest, TestReload, SuggestComponentTest, TestCoreContainer, SynonymTokenizerTest, TestNRTOpen, FileUtilsTest, AliasIntegrationTest, TestSolrJ, ShardSplitTest, TestSolr4Spatial, TestPhraseSuggestions, TestDistribDocBasedVersion, TestNoOpRegenerator, JSONWriterTest, CustomCollectionTest, TestSolrXmlPersistor, TestNumberUtils, HdfsCollectionsAPIDistributedZkTest, TestSerializedLuceneMatchVersion, TestValueSourceCache, TestHashPartitioner, UpdateRequestProcessorFactoryTest, TestSchemaNameResource, UniqFieldsUpdateProcessorFactoryTest, TestStandardQParsers, TestPseudoReturnFields, DistributedSpellCheckComponentTest, SpellCheckCollatorTest, DeleteReplicaTest, BasicFunctionalityTest, TestFastWriter, TestMergePolicyConfig, ParsingFieldUpdateProcessorsTest, QueryResultKeyTest, CacheHeaderTest, SuggesterTest, WordBreakSolrSpellCheckerTest, TestLazyCores, TestSurroundQueryParser, TestSort, TestRecoveryHdfs, CachingDirectoryFactoryTest, TestBinaryResponseWriter, UpdateParamsTest, TestRecovery, TestInfoStreamLogging, AbstractAnalyticsStatsTest, StatelessScriptUpdateProcessorFactoryTest, OverseerCollectionProcessorTest, TestSolrQueryParser, TestReversedWildcardFilterFactory, SpellCheckComponentTest, BasicDistributedZkTest, TestNonNRTOpen, TestStressReorder, TestHighFrequencyDictionaryFactory, ResponseLogComponentTest, TestQuerySenderListener, CoreAdminCreateDiscoverTest, OverseerRolesTest, TestFiltering, TestAnalyzeInfixSuggestions, CursorPagingTest, TestFieldCollectionResource, DisMaxRequestHandlerTest, TestCSVResponseWriter, TestDocumentBuilder, ZkControllerTest, AlternateDirectoryTest, TestMaxScoreQueryParser, BasicDistributedZk2Test, TestFastLRUCache, TestDFRSimilarityFactory, OutputWriterTest, DocValuesMissingTest, TestCollapseQParserPlugin, TestQueryTypes, TestFieldTypeResource, TestHighlightDedupGrouping, TestSearchPerf, SolrIndexConfigTest, UnloadDistributedZkTest, TestPerFieldSimilarity, TestSchemaVersionResource, ConvertedLegacyTest, BadCopyFieldTest, StandardRequestHandlerTest, QueryElevationComponentTest, DebugComponentTest, TestDistributedGrouping, TestRandomFaceting, TestBM25SimilarityFactory, DocValuesTest, InfoHandlerTest, RequiredFieldsTest, DefaultValueUpdateProcessorTest, TestDefaultSimilarityFactory, NumericFieldsTest, TestConfig, TestCoreDiscovery, TestCollationField, AddBlockUpdateTest, SuggesterWFSTTest, TestJmxMonitoredMap, BinaryUpdateRequestHandlerTest, BasicZkTest, TestReloadAndDeleteDocs, AssignTest, DirectUpdateHandlerOptimizeTest, RAMDirectoryFactoryTest, HdfsBasicDistributedZk2Test, TestMultiCoreConfBootstrap, SolrInfoMBeanTest, CursorMarkTest, TestQuerySenderNoQuery, TestCodecSupport, PeerSyncTest, PluginInfoTest, TestDefaultSearchFieldResource, TestTrie, EnumFieldTest, TestWordDelimiterFilterFactory, PreAnalyzedFieldTest, TestPHPSerializedResponseWriter, HdfsChaosMonkeySafeLeaderTest, ZkSolrClientTest, TestDynamicFieldCollectionResource, UUIDFieldTest, SuggesterFSTTest, SortByFunctionTest, DistributedQueryElevationComponentTest, TestRandomMergePolicy, NoCacheHeaderTest, FileBasedSpellCheckerTest, PingRequestHandlerTest, MultiTermTest, TestJmxIntegration, TestStressLucene, QueryFacetTest, TestZkChroot, RegexBoostProcessorTest, TestSchemaResource, DistanceFunctionTest, DateFieldTest, ResourceLoaderTest, SimpleFacetsTest, RangeFacetTest, HdfsBasicDistributedZkTest, DateMathParserTest, SpatialFilterTest, TestSystemIdResolver, TestShardHandlerFactory, MinimalSchemaTest, CopyFieldTest, SearchHandlerTest, TestCustomSort, SolrRequestParserTest, TestClassNameShortening, XmlUpdateRequestHandlerTest, TestIndexingPerformance, ChaosMonkeySafeLeaderTest, CollectionsAPIDistributedZkTest, SyncSliceTest, OpenCloseCoreStressTest, OverseerTest]
   [junit4] Completed in 38.18s, 8 tests, 1 error <<< FAILURES!

[...truncated 572 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:453: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:433: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:39: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-targets.xml:37: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build.xml:189: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\common-build.xml:491: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:1268: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:901: There were test failures: 372 suites, 1604 tests, 1 error, 40 ignored (12 assumptions)

Total time: 94 minutes 19 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_51 -client -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.8.0-fcs-b129) - Build # 3791 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/3791/
Java: 64bit/jdk1.8.0-fcs-b129 -XX:+UseCompressedOops -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.cloud.ShardSplitTest.testDistribSearch

Error Message:
Wrong doc count on shard1_0 expected:<122> but was:<121>

Stack Trace:
java.lang.AssertionError: Wrong doc count on shard1_0 expected:<122> but was:<121>
	at __randomizedtesting.SeedInfo.seed([87906A451C50BB29:676E45D6B0FDB15]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.cloud.ShardSplitTest.checkDocCountsAndShardStates(ShardSplitTest.java:477)
	at org.apache.solr.cloud.ShardSplitTest.splitByUniqueKeyTest(ShardSplitTest.java:249)
	at org.apache.solr.cloud.ShardSplitTest.doTest(ShardSplitTest.java:113)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 10259 lines...]
   [junit4] Suite: org.apache.solr.cloud.ShardSplitTest
   [junit4]   2> 944017 T4399 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 944017 T4399 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /quce/
   [junit4]   2> 944021 T4399 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\solrtest-ShardSplitTest-1392685238539
   [junit4]   2> 944024 T4399 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 944024 T4400 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 944095 T4399 oasc.ZkTestServer.run start zk server on port:49977
   [junit4]   2> 944098 T4399 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 944103 T4406 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5ac02070 name:ZooKeeperConnection Watcher:127.0.0.1:49977 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 944104 T4399 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 944104 T4399 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 944110 T4401 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x144428249110000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 944112 T4399 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 944115 T4408 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10fc55c2 name:ZooKeeperConnection Watcher:127.0.0.1:49977/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 944115 T4399 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 944115 T4399 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 944119 T4399 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 944122 T4399 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 944124 T4399 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 944128 T4399 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 944128 T4399 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 944133 T4399 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 944133 T4399 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 944137 T4399 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 944137 T4399 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 944142 T4399 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 944142 T4399 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 944147 T4399 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 944147 T4399 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 944153 T4399 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 944153 T4399 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 944156 T4399 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 944156 T4399 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 944160 T4399 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 944160 T4399 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 944165 T4399 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 944165 T4399 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 944169 T4399 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 944169 T4399 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 944172 T4399 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 944173 T4399 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 944177 T4401 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x144428249110001, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 944178 T4399 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 944181 T4410 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13adf288 name:ZooKeeperConnection Watcher:127.0.0.1:49977/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 944181 T4399 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 944184 T4401 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x144428249110002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 944556 T4399 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 944563 T4399 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 944568 T4399 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:49987
   [junit4]   2> 944571 T4399 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 944576 T4399 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 944576 T4399 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: .\org.apache.solr.cloud.ShardSplitTest-controljetty-1392685238700
   [junit4]   2> 944576 T4399 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '.\org.apache.solr.cloud.ShardSplitTest-controljetty-1392685238700\'
   [junit4]   2> 944601 T4399 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ShardSplitTest-controljetty-1392685238700\solr.xml
   [junit4]   2> 944608 T4399 oasc.CoreContainer.<init> New CoreContainer 1532019868
   [junit4]   2> 944608 T4399 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=.\org.apache.solr.cloud.ShardSplitTest-controljetty-1392685238700\]
   [junit4]   2> 944610 T4399 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 944610 T4399 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 944610 T4399 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 944610 T4399 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 944610 T4399 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 944610 T4399 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 944610 T4399 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 944611 T4399 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 944611 T4399 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 944619 T4399 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 944619 T4399 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 944619 T4399 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 944619 T4399 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49977/solr
   [junit4]   2> 944619 T4399 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 944622 T4399 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 944629 T4421 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7eb4c787 name:ZooKeeperConnection Watcher:127.0.0.1:49977 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 944629 T4399 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 944631 T4401 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x144428249110003, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 944631 T4399 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 944634 T4423 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69acd5ec name:ZooKeeperConnection Watcher:127.0.0.1:49977/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 944634 T4399 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 944636 T4399 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 944640 T4399 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 944645 T4399 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 944647 T4399 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49987_quce
   [junit4]   2> 944649 T4399 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49987_quce
   [junit4]   2> 944652 T4399 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 944654 T4399 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 944660 T4399 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:49987_quce
   [junit4]   2> 944660 T4399 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 944663 T4399 oasc.Overseer.start Overseer (id=91271019793285124-127.0.0.1:49987_quce-n_0000000000) starting
   [junit4]   2> 944667 T4399 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 944672 T4425 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 944674 T4399 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 944674 T4425 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 944676 T4399 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 944678 T4399 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 944683 T4424 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 944685 T4426 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 944685 T4426 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 944686 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 944686 T4426 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 944687 T4424 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 944688 T4424 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:49987/quce",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49987_quce",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 944688 T4424 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 944688 T4424 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 944690 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 944692 T4423 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 945401 T4426 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 945402 T4426 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: .\org.apache.solr.cloud.ShardSplitTest-controljetty-1392685238700\collection1
   [junit4]   2> 945402 T4426 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 945403 T4426 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 945403 T4426 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 945404 T4426 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 945404 T4426 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '.\org.apache.solr.cloud.ShardSplitTest-controljetty-1392685238700\collection1\'
   [junit4]   2> 945407 T4426 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-controljetty-1392685238700/collection1/lib/classes/' to classloader
   [junit4]   2> 945407 T4426 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-controljetty-1392685238700/collection1/lib/README' to classloader
   [junit4]   2> 945438 T4426 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 945444 T4426 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 945446 T4426 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 945452 T4426 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 945533 T4426 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 945534 T4426 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 945534 T4426 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 945538 T4426 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 945564 T4426 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 945564 T4426 oasc.SolrCore.<init> [collection1] Opening new SolrCore at .\org.apache.solr.cloud.ShardSplitTest-controljetty-1392685238700\collection1\, dataDir=.\org.apache.solr.cloud.ShardSplitTest-1392685238537/control/data\
   [junit4]   2> 945564 T4426 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@206d8fd1
   [junit4]   2> 945565 T4426 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.ShardSplitTest-1392685238537/control/data\
   [junit4]   2> 945565 T4426 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=.\org.apache.solr.cloud.ShardSplitTest-1392685238537/control/data\index/
   [junit4]   2> 945565 T4426 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '.\org.apache.solr.cloud.ShardSplitTest-1392685238537\control\data\index' doesn't exist. Creating new index...
   [junit4]   2> 945565 T4426 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.ShardSplitTest-1392685238537/control/data\index
   [junit4]   2> 945566 T4426 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=5, maxMergeAtOnceExplicit=6, maxMergedSegmentMB=13.474609375, floorSegmentMB=1.666015625, forceMergeDeletesPctAllowed=11.964719886809188, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 945566 T4426 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@429d64e1 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4a3ceb26),segFN=segments_1,generation=1}
   [junit4]   2> 945567 T4426 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 945568 T4426 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 945568 T4426 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 945569 T4426 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 945569 T4426 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 945569 T4426 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 945570 T4426 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 945570 T4426 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 945571 T4426 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 945571 T4426 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 945571 T4426 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 945571 T4426 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 945572 T4426 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 945572 T4426 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 945573 T4426 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 945573 T4426 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 945574 T4426 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 945574 T4426 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 945581 T4426 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 945584 T4426 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 945584 T4426 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 945584 T4426 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=47, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=47.9228515625, floorSegmentMB=1.033203125, forceMergeDeletesPctAllowed=17.023393992292856, segmentsPerTier=28.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6037720933983147
   [junit4]   2> 945585 T4426 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@429d64e1 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4a3ceb26),segFN=segments_1,generation=1}
   [junit4]   2> 945585 T4426 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 945586 T4426 oass.SolrIndexSearcher.<init> Opening Searcher@56eb3a2f[collection1] main
   [junit4]   2> 945587 T4427 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@56eb3a2f[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 945588 T4426 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 945589 T4399 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 945589 T4430 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:49987/quce collection:control_collection shard:shard1
   [junit4]   2> 945589 T4399 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 945591 T4430 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 945598 T4399 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 945601 T4432 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@312e8314 name:ZooKeeperConnection Watcher:127.0.0.1:49977/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 945601 T4399 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 945601 T4430 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 945603 T4399 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 945605 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 945606 T4430 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 945606 T4430 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C797 name=collection1 org.apache.solr.core.SolrCore@72c2ac02 url=https://127.0.0.1:49987/quce/collection1 node=127.0.0.1:49987_quce C797_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:49987/quce, core=collection1, node_name=127.0.0.1:49987_quce}
   [junit4]   2> 945606 T4430 C797 P49987 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:49987/quce/collection1/
   [junit4]   2> 945606 T4430 C797 P49987 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 945606 T4430 C797 P49987 oasc.SyncStrategy.syncToMe https://127.0.0.1:49987/quce/collection1/ has no replicas
   [junit4]   2> 945606 T4430 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:49987/quce/collection1/ shard1
   [junit4]   2> 945606 T4430 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 945607 T4399 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 945607 T4424 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 945613 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 945618 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 945623 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 945705 T4432 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 945706 T4423 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 945739 T4430 oasc.ZkController.register We are https://127.0.0.1:49987/quce/collection1/ and leader is https://127.0.0.1:49987/quce/collection1/
   [junit4]   2> 945739 T4430 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:49987/quce
   [junit4]   2> 945739 T4430 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 945739 T4430 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 945739 T4430 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 945742 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 945742 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 945742 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 945744 T4430 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 945745 T4424 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 945745 T4424 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:49987/quce",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49987_quce",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 945749 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 945831 T4432 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 945832 T4423 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 946031 T4399 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 946031 T4399 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 946040 T4399 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 946041 T4399 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:49999
   [junit4]   2> 946045 T4399 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 946045 T4399 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 946046 T4399 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: .\org.apache.solr.cloud.ShardSplitTest-jetty1-1392685240123
   [junit4]   2> 946046 T4399 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '.\org.apache.solr.cloud.ShardSplitTest-jetty1-1392685240123\'
   [junit4]   2> 946067 T4399 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ShardSplitTest-jetty1-1392685240123\solr.xml
   [junit4]   2> 946082 T4399 oasc.CoreContainer.<init> New CoreContainer 751204352
   [junit4]   2> 946083 T4399 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=.\org.apache.solr.cloud.ShardSplitTest-jetty1-1392685240123\]
   [junit4]   2> 946083 T4399 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 946083 T4399 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 946083 T4399 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 946083 T4399 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 946084 T4399 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 946084 T4399 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 946084 T4399 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 946084 T4399 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 946085 T4399 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 946093 T4399 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 946093 T4399 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 946093 T4399 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 946093 T4399 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49977/solr
   [junit4]   2> 946093 T4399 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 946096 T4399 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 946102 T4443 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@31f8dba4 name:ZooKeeperConnection Watcher:127.0.0.1:49977 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 946102 T4399 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 946105 T4399 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 946109 T4445 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d0daafc name:ZooKeeperConnection Watcher:127.0.0.1:49977/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 946109 T4399 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 946116 T4399 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 946901 T4399 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49999_quce
   [junit4]   2> 946904 T4399 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49999_quce
   [junit4]   2> 946909 T4432 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 946909 T4445 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 946909 T4423 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 946916 T4446 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 946916 T4446 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 946918 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946918 T4446 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 946918 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946918 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 946920 T4424 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 946920 T4424 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:49999/quce",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49999_quce",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 946920 T4424 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 946920 T4424 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 946924 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 947006 T4445 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 947006 T4432 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 947007 T4423 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 947699 T4446 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 947699 T4446 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: .\org.apache.solr.cloud.ShardSplitTest-jetty1-1392685240123\collection1
   [junit4]   2> 947699 T4446 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 947702 T4446 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 947702 T4446 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 947705 T4446 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 947705 T4446 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '.\org.apache.solr.cloud.ShardSplitTest-jetty1-1392685240123\collection1\'
   [junit4]   2> 947705 T4446 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty1-1392685240123/collection1/lib/classes/' to classloader
   [junit4]   2> 947707 T4446 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty1-1392685240123/collection1/lib/README' to classloader
   [junit4]   2> 947734 T4446 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 947741 T4446 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 947743 T4446 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 947747 T4446 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 947828 T4446 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 947829 T4446 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 947830 T4446 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 947834 T4446 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 947863 T4446 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 947863 T4446 oasc.SolrCore.<init> [collection1] Opening new SolrCore at .\org.apache.solr.cloud.ShardSplitTest-jetty1-1392685240123\collection1\, dataDir=.\org.apache.solr.cloud.ShardSplitTest-1392685238537/jetty1\
   [junit4]   2> 947863 T4446 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@206d8fd1
   [junit4]   2> 947863 T4446 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.ShardSplitTest-1392685238537/jetty1\
   [junit4]   2> 947864 T4446 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=.\org.apache.solr.cloud.ShardSplitTest-1392685238537/jetty1\index/
   [junit4]   2> 947864 T4446 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '.\org.apache.solr.cloud.ShardSplitTest-1392685238537\jetty1\index' doesn't exist. Creating new index...
   [junit4]   2> 947864 T4446 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.ShardSplitTest-1392685238537/jetty1\index
   [junit4]   2> 947864 T4446 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=5, maxMergeAtOnceExplicit=6, maxMergedSegmentMB=13.474609375, floorSegmentMB=1.666015625, forceMergeDeletesPctAllowed=11.964719886809188, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 947865 T4446 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1249fd5d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@552521eb),segFN=segments_1,generation=1}
   [junit4]   2> 947865 T4446 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 947867 T4446 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 947868 T4446 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 947868 T4446 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 947868 T4446 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 947869 T4446 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 947869 T4446 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 947869 T4446 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 947869 T4446 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 947870 T4446 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 947870 T4446 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 947870 T4446 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 947870 T4446 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 947870 T4446 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 947870 T4446 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 947871 T4446 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 947872 T4446 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 947872 T4446 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 947876 T4446 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 947879 T4446 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 947879 T4446 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 947880 T4446 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=47, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=47.9228515625, floorSegmentMB=1.033203125, forceMergeDeletesPctAllowed=17.023393992292856, segmentsPerTier=28.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6037720933983147
   [junit4]   2> 947880 T4446 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1249fd5d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@552521eb),segFN=segments_1,generation=1}
   [junit4]   2> 947880 T4446 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 947880 T4446 oass.SolrIndexSearcher.<init> Opening Searcher@21efc04c[collection1] main
   [junit4]   2> 947886 T4446 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 947888 T4399 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 947888 T4399 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 947911 T4447 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@21efc04c[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 947912 T4450 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:49999/quce collection:collection1 shard:shard2
   [junit4]   2> 947914 T4450 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 947931 T4450 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 947934 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 947934 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 947934 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 947935 T4450 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 947935 T4450 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C798 name=collection1 org.apache.solr.core.SolrCore@3130f16d url=https://127.0.0.1:49999/quce/collection1 node=127.0.0.1:49999_quce C798_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:49999/quce, core=collection1, node_name=127.0.0.1:49999_quce}
   [junit4]   2> 947935 T4450 C798 P49999 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:49999/quce/collection1/
   [junit4]   2> 947935 T4450 C798 P49999 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 947935 T4450 C798 P49999 oasc.SyncStrategy.syncToMe https://127.0.0.1:49999/quce/collection1/ has no replicas
   [junit4]   2> 947935 T4450 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:49999/quce/collection1/ shard2
   [junit4]   2> 947935 T4450 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 947938 T4424 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 947944 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 947947 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 947953 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 948035 T4432 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 948036 T4423 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 948036 T4445 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 948067 T4450 oasc.ZkController.register We are https://127.0.0.1:49999/quce/collection1/ and leader is https://127.0.0.1:49999/quce/collection1/
   [junit4]   2> 948067 T4450 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:49999/quce
   [junit4]   2> 948067 T4450 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 948067 T4450 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 948067 T4450 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 948069 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 948069 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 948069 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 948069 T4450 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 948072 T4424 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 948072 T4424 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:49999/quce",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49999_quce",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 948076 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 948163 T4445 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 948163 T4432 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 948163 T4423 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 948308 T4399 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 948309 T4399 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 948316 T4399 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 948317 T4399 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:50008
   [junit4]   2> 948321 T4399 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 948321 T4399 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 948321 T4399 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ShardSplitTest-jetty2-1392685242408
   [junit4]   2> 948321 T4399 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ShardSplitTest-jetty2-1392685242408\'
   [junit4]   2> 948344 T4399 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ShardSplitTest-jetty2-1392685242408\solr.xml
   [junit4]   2> 948354 T4399 oasc.CoreContainer.<init> New CoreContainer 2076811545
   [junit4]   2> 948354 T4399 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ShardSplitTest-jetty2-1392685242408\]
   [junit4]   2> 948355 T4399 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 948355 T4399 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 948355 T4399 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 948355 T4399 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 948355 T4399 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 948355 T4399 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 948355 T4399 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 948355 T4399 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 948355 T4399 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 948363 T4399 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 948363 T4399 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 948363 T4399 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 948363 T4399 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49977/solr
   [junit4]   2> 948363 T4399 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 948365 T4399 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 948370 T4461 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@725d5071 name:ZooKeeperConnection Watcher:127.0.0.1:49977 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 948372 T4399 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 948375 T4401 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x144428249110008, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 948376 T4399 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 948379 T4463 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@41328347 name:ZooKeeperConnection Watcher:127.0.0.1:49977/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 948379 T4399 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 948385 T4399 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 949170 T4399 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50008_quce
   [junit4]   2> 949172 T4399 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50008_quce
   [junit4]   2> 949177 T4445 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 949177 T4432 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 949177 T4423 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 949177 T4463 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 949185 T4464 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 949185 T4464 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 949187 T4464 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 949187 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 949187 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 949187 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 949190 T4424 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 949190 T4424 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50008/quce",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50008_quce",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 949190 T4424 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 949190 T4424 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 949193 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 949274 T4423 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 949274 T4463 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 949274 T4445 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 949275 T4432 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 949969 T4464 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 949969 T4464 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ShardSplitTest-jetty2-1392685242408\collection1
   [junit4]   2> 949969 T4464 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 949971 T4464 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 949971 T4464 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 949976 T4464 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 949976 T4464 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ShardSplitTest-jetty2-1392685242408\collection1\'
   [junit4]   2> 949980 T4464 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty2-1392685242408/collection1/lib/classes/' to classloader
   [junit4]   2> 949982 T4464 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty2-1392685242408/collection1/lib/README' to classloader
   [junit4]   2> 950006 T4464 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 950012 T4464 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 950014 T4464 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 950018 T4464 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 950101 T4464 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 950102 T4464 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 950104 T4464 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 950110 T4464 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 950140 T4464 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 950140 T4464 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ShardSplitTest-jetty2-1392685242408\collection1\, dataDir=.\org.apache.solr.cloud.ShardSplitTest-1392685238537/jetty2\
   [junit4]   2> 950141 T4464 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@206d8fd1
   [junit4]   2> 950142 T4464 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.ShardSplitTest-1392685238537/jetty2\
   [junit4]   2> 950142 T4464 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=.\org.apache.solr.cloud.ShardSplitTest-1392685238537/jetty2\index/
   [junit4]   2> 950142 T4464 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '.\org.apache.solr.cloud.ShardSplitTest-1392685238537\jetty2\index' doesn't exist. Creating new index...
   [junit4]   2> 950143 T4464 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.ShardSplitTest-1392685238537/jetty2\index
   [junit4]   2> 950143 T4464 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=5, maxMergeAtOnceExplicit=6, maxMergedSegmentMB=13.474609375, floorSegmentMB=1.666015625, forceMergeDeletesPctAllowed=11.964719886809188, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 950143 T4464 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@494079a6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5ee9e5eb),segFN=segments_1,generation=1}
   [junit4]   2> 950143 T4464 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 950145 T4464 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 950146 T4464 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 950146 T4464 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 950146 T4464 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 950146 T4464 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 950146 T4464 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 950147 T4464 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 950147 T4464 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 950147 T4464 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 950148 T4464 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 950149 T4464 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 950149 T4464 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 950149 T4464 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 950149 T4464 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 950149 T4464 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 950149 T4464 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 950151 T4464 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 950155 T4464 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 950158 T4464 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 950159 T4464 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 950159 T4464 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=47, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=47.9228515625, floorSegmentMB=1.033203125, forceMergeDeletesPctAllowed=17.023393992292856, segmentsPerTier=28.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6037720933983147
   [junit4]   2> 950159 T4464 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@494079a6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5ee9e5eb),segFN=segments_1,generation=1}
   [junit4]   2> 950159 T4464 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 950159 T4464 oass.SolrIndexSearcher.<init> Opening Searcher@1bd354c3[collection1] main
   [junit4]   2> 950162 T4465 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1bd354c3[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 950164 T4464 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 950165 T4399 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 950165 T4399 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 950165 T4468 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:50008/quce collection:collection1 shard:shard1
   [junit4]   2> 950167 T4468 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 950307 T4468 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 950539 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 950539 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 950539 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 950542 T4468 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 950542 T4468 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C799 name=collection1 org.apache.solr.core.SolrCore@75ca42ee url=https://127.0.0.1:50008/quce/collection1 node=127.0.0.1:50008_quce C799_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:50008/quce, core=collection1, node_name=127.0.0.1:50008_quce}
   [junit4]   2> 950542 T4468 C799 P50008 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:50008/quce/collection1/
   [junit4]   2> 950542 T4468 C799 P50008 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 950542 T4468 C799 P50008 oasc.SyncStrategy.syncToMe https://127.0.0.1:50008/quce/collection1/ has no replicas
   [junit4]   2> 950542 T4468 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:50008/quce/collection1/ shard1
   [junit4]   2> 950542 T4468 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 950545 T4424 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 950552 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 950555 T4399 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 950555 T4399 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 950556 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 950562 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 950563 T4399 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 950565 T4399 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:50017
   [junit4]   2> 950569 T4399 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 950569 T4399 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 950569 T4399 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ShardSplitTest-jetty3-1392685244685
   [junit4]   2> 950570 T4399 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ShardSplitTest-jetty3-1392685244685\'
   [junit4]   2> 950591 T4399 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ShardSplitTest-jetty3-1392685244685\solr.xml
   [junit4]   2> 950602 T4399 oasc.CoreContainer.<init> New CoreContainer 1268343145
   [junit4]   2> 950602 T4399 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ShardSplitTest-jetty3-1392685244685\]
   [junit4]   2> 950603 T4399 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 950603 T4399 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 950603 T4399 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 950603 T4399 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 950603 T4399 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 950604 T4399 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 950604 T4399 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 950604 T4399 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 950604 T4399 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 950613 T4399 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 950613 T4399 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 950613 T4399 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 950613 T4399 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49977/solr
   [junit4]   2> 950615 T4399 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 950616 T4399 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 950620 T4479 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@26333e2f name:ZooKeeperConnection Watcher:127.0.0.1:49977 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 950620 T4399 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 950625 T4399 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 950628 T4481 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6c4f3707 name:ZooKeeperConnection Watcher:127.0.0.1:49977/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 950629 T4399 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 950635 T4399 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 950642 T4432 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 950644 T4423 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 950644 T4463 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 950645 T4445 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 950645 T4481 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 950676 T4468 oasc.ZkController.register We are https://127.0.0.1:50008/quce/collection1/ and leader is https://127.0.0.1:50008/quce/collection1/
   [junit4]   2> 950676 T4468 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:50008/quce
   [junit4]   2> 950676 T4468 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 950676 T4468 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 950676 T4468 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 950678 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 950679 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 950679 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 950679 T4468 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 950681 T4424 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 950682 T4424 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50008/quce",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50008_quce",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 950687 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 950768 T4445 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 950769 T4432 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 950769 T4481 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 950769 T4463 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 950769 T4423 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 951420 T4399 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50017_quce
   [junit4]   2> 951423 T4399 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50017_quce
   [junit4]   2> 951427 T4445 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 951427 T4432 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 951427 T4423 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 951427 T4481 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 951427 T4463 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 951436 T4482 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 951436 T4482 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 951438 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 951438 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 951438 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 951438 T4482 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 951440 T4424 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 951441 T4424 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50017/quce",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50017_quce",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 951441 T4424 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 951441 T4424 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 951445 T4423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 951526 T4445 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 951527 T4432 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 951527 T4481 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 951527 T4463 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 951527 T4423 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 952220 T4482 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 952220 T4482 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ShardSplitTest-jetty3-1392685244685\collection1
   [junit4]   2> 952220 T4482 oasc.ZkController.createCollectionZkNode Check 

[...truncated too long message...]

 [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 1000125 T4500 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1000233 T4537 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1404ee13 name:ZooKeeperConnection Watcher:127.0.0.1:49977/solr got event WatchedEvent state:Disconnected type:None path:null path:null type:None
   [junit4]   2> 1000233 T4537 oascc.ConnectionManager.process Client->ZooKeeper status change trigger but we are already closed
   [junit4]   2> 1000233 T4399 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 49987
   [junit4]   2> 1000276 T4399 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 1000276 T4399 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:49977 49977
   [junit4]   1>               "range":"80000000-dfffffff",
   [junit4]   1>               "state":"recovery",
   [junit4]   1>               "replicas":{
   [junit4]   1>                 "core_node5":{
   [junit4]   1>                   "state":"active",
   [junit4]   1>                   "base_url":"https://127.0.0.1:50008/quce",
   [junit4]   1>                   "core":"collection1_shard1_0_replica1",
   [junit4]   1>                   "node_name":"127.0.0.1:50008_quce",
   [junit4]   1>                   "leader":"true"},
   [junit4]   1>                 "core_node7":{
   [junit4]   1>                   "state":"recovering",
   [junit4]   1>                   "base_url":"https://127.0.0.1:50027/quce",
   [junit4]   1>                   "core":"collection1_shard1_0_replica2",
   [junit4]   1>                   "node_name":"127.0.0.1:50027_quce"}}},
   [junit4]   1>             "shard1_1":{
   [junit4]   1>               "parent":"shard1",
   [junit4]   1>               "range":"e0000000-ffffffff",
   [junit4]   1>               "state":"recovery",
   [junit4]   1>               "replicas":{
   [junit4]   1>                 "core_node6":{
   [junit4]   1>                   "state":"active",
   [junit4]   1>                   "base_url":"https://127.0.0.1:50008/quce",
   [junit4]   1>                   "core":"collection1_shard1_1_replica1",
   [junit4]   1>                   "node_name":"127.0.0.1:50008_quce",
   [junit4]   1>                   "leader":"true"},
   [junit4]   1>                 "core_node8":{
   [junit4]   1>                   "state":"recovering",
   [junit4]   1>                   "base_url":"https://127.0.0.1:50027/quce",
   [junit4]   1>                   "core":"collection1_shard1_1_replica2",
   [junit4]   1>                   "node_name":"127.0.0.1:50027_quce"}}}},
   [junit4]   1>           "router":{"name":"compositeId"},
   [junit4]   1>           "autoCreated":"true",
   [junit4]   1>           "maxShardsPerNode":"1"}}
   [junit4]   1>   /solr/clusterprops.json (0)
   [junit4]   1>   DATA:
   [junit4]   1>       {"urlScheme":"https"}
   [junit4]   1> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ShardSplitTest -Dtests.method=testDistribSearch -Dtests.seed=87906A451C50BB29 -Dtests.slow=true -Dtests.locale=ar_SY -Dtests.timezone=Australia/Brisbane -Dtests.file.encoding=Cp1252
   [junit4] FAILURE 56.3s | ShardSplitTest.testDistribSearch <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Wrong doc count on shard1_0 expected:<122> but was:<121>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([87906A451C50BB29:676E45D6B0FDB15]:0)
   [junit4]    > 	at org.apache.solr.cloud.ShardSplitTest.checkDocCountsAndShardStates(ShardSplitTest.java:477)
   [junit4]    > 	at org.apache.solr.cloud.ShardSplitTest.splitByUniqueKeyTest(ShardSplitTest.java:249)
   [junit4]    > 	at org.apache.solr.cloud.ShardSplitTest.doTest(ShardSplitTest.java:113)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 1000289 T4399 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 56285 T4398 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Lucene46: {_version_=PostingsFormat(name=Memory doPackFST= false), n_ti=PostingsFormat(name=Memory doPackFST= false), id=Lucene41(blocksize=128)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=ar_SY, timezone=Australia/Brisbane
   [junit4]   2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.8.0 (64-bit)/cpus=2,threads=1,free=85859552,total=261095424
   [junit4]   2> NOTE: All tests run in this JVM: [AbstractAnalyticsStatsTest, CoreAdminCreateDiscoverTest, SuggesterTest, TestSolrDeletionPolicy1, TestLFUCache, ZkSolrClientTest, TestReversedWildcardFilterFactory, HdfsUnloadDistributedZkTest, TestSchemaNameResource, ZkControllerTest, TestIndexingPerformance, TestLMJelinekMercerSimilarityFactory, TestFoldingMultitermQuery, TestFieldTypeResource, StandardRequestHandlerTest, DirectSolrSpellCheckerTest, TestSimpleQParserPlugin, MoreLikeThisHandlerTest, TestXIncludeConfig, OverseerCollectionProcessorTest, TestSerializedLuceneMatchVersion, TestReplicationHandler, DeleteReplicaTest, ConnectionManagerTest, TestIndexSearcher, StressHdfsTest, TestAtomicUpdateErrorCases, QueryFacetTest, TestReloadAndDeleteDocs, HighlighterTest, ShowFileRequestHandlerTest, TestRandomMergePolicy, SearchHandlerTest, TestInfoStreamLogging, TestLazyCores, DateMathParserTest, DistributedSuggestComponentTest, TestSolrQueryParserDefaultOperatorResource, FieldFacetExtrasTest, TestCoreContainer, TestFileDictionaryLookup, TestDefaultSimilarityFactory, TestCloudManagedSchemaAddField, TestRangeQuery, CoreMergeIndexesAdminHandlerTest, OverseerTest, ModifyConfFileTest, TestRandomFaceting, AlternateDirectoryTest, AddBlockUpdateTest, TestManagedSchema, RequestHandlersTest, PathHierarchyTokenizerFactoryTest, TestJoin, QueryParsingTest, TestStressVersions, FastVectorHighlighterTest, TestCopyFieldCollectionResource, SpellingQueryConverterTest, SpatialFilterTest, TestSweetSpotSimilarityFactory, TestSolr4Spatial, MultiTermTest, SpellCheckCollatorTest, TestStressReorder, BadIndexSchemaTest, TestExtendedDismaxParser, DateFieldTest, SolrIndexSplitterTest, UniqFieldsUpdateProcessorFactoryTest, EchoParamsTest, TestBinaryResponseWriter, CursorPagingTest, ChaosMonkeyNothingIsSafeTest, BasicFunctionalityTest, TestLRUCache, TestSort, PreAnalyzedUpdateProcessorTest, CachingDirectoryFactoryTest, FileBasedSpellCheckerTest, HdfsChaosMonkeySafeLeaderTest, LukeRequestHandlerTest, TestSchemaSimilarityResource, DirectSolrConnectionTest, TestUniqueKeyFieldResource, TestFieldCollectionResource, TestSchemaResource, TestOmitPositions, PrimUtilsTest, TestDocumentBuilder, TestCustomSort, TestSolrXmlPersistor, HdfsSyncSliceTest, TestGroupingSearch, ShardRoutingTest, TestPerFieldSimilarity, TestStressRecovery, TriLevelCompositeIdRoutingTest, SolrCoreCheckLockOnStartupTest, TestNonNRTOpen, UpdateParamsTest, JsonLoaderTest, TestRTGBase, FieldMutatingUpdateProcessorTest, HdfsCollectionsAPIDistributedZkTest, CacheHeaderTest, TestFaceting, TestShardHandlerFactory, ConvertedLegacyTest, HdfsRecoveryZkTest, DistribCursorPagingTest, TestFreeTextSuggestions, TestManagedSchemaFieldResource, ShardSplitTest]
   [junit4] Completed in 56.94s, 1 test, 1 failure <<< FAILURES!

[...truncated 781 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:453: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:433: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:39: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-targets.xml:37: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build.xml:189: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\common-build.xml:491: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:1268: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:901: There were test failures: 372 suites, 1604 tests, 1 failure, 40 ignored (12 assumptions)

Total time: 68 minutes 2 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0-fcs-b129 -XX:+UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Re: [JENKINS] Lucene-Solr-trunk-Windows (32bit/jdk1.8.0-fcs-b129) - Build # 3790 - Still Failing!

Posted by Mark Miller <ma...@gmail.com>.
On first glance, looks like a lookup using SSL. Or super slowness. Slowness is a common thing to see so far.

- Mark

http://about.me/markrmiller

On Feb 17, 2014, at 1:55 PM, Policeman Jenkins Server <je...@thetaphi.de> wrote:

> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/3790/
> Java: 32bit/jdk1.8.0-fcs-b129 -server -XX:+UseSerialGC
> 
> 2 tests failed.
> REGRESSION:  org.apache.solr.cloud.RecoveryZkTest.testDistribSearch
> 
> Error Message:
> Test abandoned because suite timeout was reached.
> 
> Stack Trace:
> java.lang.Exception: Test abandoned because suite timeout was reached.
> 	at __randomizedtesting.SeedInfo.seed([41014895D1172F7D]:0)
> 
> 
> FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest
> 
> Error Message:
> Suite timeout exceeded (>= 7200000 msec).
> 
> Stack Trace:
> java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
> 	at __randomizedtesting.SeedInfo.seed([41014895D1172F7D]:0)
> 
> 
> 
> 
> Build Log:
> [...truncated 10038 lines...]
>   [junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
>   [junit4]   2> 73152 T398 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
>   [junit4]   2> 73153 T398 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /sv/
>   [junit4]   2> 73156 T398 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
>   [junit4]   2> Creating dataDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\solrtest-RecoveryZkTest-1392655614516
>   [junit4]   2> 73158 T398 oasc.ZkTestServer.run STARTING ZK TEST SERVER
>   [junit4]   2> 73159 T399 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
>   [junit4]   2> 73263 T398 oasc.ZkTestServer.run start zk server on port:64770
>   [junit4]   2> 73265 T398 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 73273 T405 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1de7949 name:ZooKeeperConnection Watcher:127.0.0.1:64770 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 73273 T398 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 73273 T398 oascc.SolrZkClient.makePath makePath: /solr
>   [junit4]   2> 73284 T398 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 73287 T407 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@184e80 name:ZooKeeperConnection Watcher:127.0.0.1:64770/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 73288 T398 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 73288 T398 oascc.SolrZkClient.makePath makePath: /collections/collection1
>   [junit4]   2> 73294 T398 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
>   [junit4]   2> 73300 T398 oascc.SolrZkClient.makePath makePath: /collections/control_collection
>   [junit4]   2> 73303 T398 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
>   [junit4]   2> 73308 T398 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
>   [junit4]   2> 73308 T398 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
>   [junit4]   2> 73317 T398 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\schema15.xml to /configs/conf1/schema.xml
>   [junit4]   2> 73317 T398 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
>   [junit4]   2> 73325 T398 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>   [junit4]   2> 73325 T398 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>   [junit4]   2> 73331 T398 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\stopwords.txt to /configs/conf1/stopwords.txt
>   [junit4]   2> 73331 T398 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
>   [junit4]   2> 73337 T398 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\protwords.txt to /configs/conf1/protwords.txt
>   [junit4]   2> 73338 T398 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
>   [junit4]   2> 73343 T398 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\currency.xml to /configs/conf1/currency.xml
>   [junit4]   2> 73343 T398 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
>   [junit4]   2> 73358 T398 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\enumsConfig.xml to /configs/conf1/enumsConfig.xml
>   [junit4]   2> 73358 T398 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
>   [junit4]   2> 73363 T398 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
>   [junit4]   2> 73363 T398 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
>   [junit4]   2> 73369 T398 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
>   [junit4]   2> 73370 T398 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
>   [junit4]   2> 73375 T398 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\old_synonyms.txt to /configs/conf1/old_synonyms.txt
>   [junit4]   2> 73376 T398 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
>   [junit4]   2> 73381 T398 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\synonyms.txt to /configs/conf1/synonyms.txt
>   [junit4]   2> 73381 T398 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
>   [junit4]   2> 73391 T398 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 73414 T409 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13b72d0 name:ZooKeeperConnection Watcher:127.0.0.1:64770/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 73414 T398 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 73419 T400 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14440be423e0002, likely client has closed socket
>   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
>   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
>   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
>   [junit4]   2> 
>   [junit4]   2> 73942 T398 oejs.Server.doStart jetty-8.1.10.v20130312
>   [junit4]   2> 73958 T398 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
>   [junit4]   2> 73969 T398 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:64780
>   [junit4]   2> 73974 T398 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>   [junit4]   2> 73975 T398 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>   [junit4]   2> 73975 T398 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: .\org.apache.solr.cloud.RecoveryZkTest-controljetty-1392655614778
>   [junit4]   2> 73975 T398 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '.\org.apache.solr.cloud.RecoveryZkTest-controljetty-1392655614778\'
>   [junit4]   2> 74013 T398 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.RecoveryZkTest-controljetty-1392655614778\solr.xml
>   [junit4]   2> 74030 T398 oasc.CoreContainer.<init> New CoreContainer 14266042
>   [junit4]   2> 74031 T398 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=.\org.apache.solr.cloud.RecoveryZkTest-controljetty-1392655614778\]
>   [junit4]   2> 74032 T398 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
>   [junit4]   2> 74033 T398 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
>   [junit4]   2> 74033 T398 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>   [junit4]   2> 74033 T398 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>   [junit4]   2> 74033 T398 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>   [junit4]   2> 74033 T398 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>   [junit4]   2> 74034 T398 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>   [junit4]   2> 74034 T398 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>   [junit4]   2> 74034 T398 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>   [junit4]   2> 74052 T398 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>   [junit4]   2> 74052 T398 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>   [junit4]   2> 74052 T398 oasc.CoreContainer.load Host Name: 127.0.0.1
>   [junit4]   2> 74052 T398 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:64770/solr
>   [junit4]   2> 74052 T398 oasc.ZkController.checkChrootPath zkHost includes chroot
>   [junit4]   2> 74056 T398 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 74068 T420 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7393ab name:ZooKeeperConnection Watcher:127.0.0.1:64770 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 74070 T398 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 74080 T398 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 74086 T422 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b2269b name:ZooKeeperConnection Watcher:127.0.0.1:64770/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 74086 T398 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 74089 T398 oascc.SolrZkClient.makePath makePath: /overseer/queue
>   [junit4]   2> 74096 T398 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
>   [junit4]   2> 74104 T398 oascc.SolrZkClient.makePath makePath: /live_nodes
>   [junit4]   2> 74109 T398 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64780_sv
>   [junit4]   2> 74111 T398 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:64780_sv
>   [junit4]   2> 74117 T398 oascc.SolrZkClient.makePath makePath: /overseer_elect
>   [junit4]   2> 74121 T398 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
>   [junit4]   2> 74130 T398 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:64780_sv
>   [junit4]   2> 74132 T398 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
>   [junit4]   2> 74135 T398 oasc.Overseer.start Overseer (id=91269078353575940-127.0.0.1:64780_sv-n_0000000000) starting
>   [junit4]   2> 74141 T398 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
>   [junit4]   2> 74148 T424 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
>   [junit4]   2> 74150 T398 oascc.SolrZkClient.makePath makePath: /clusterstate.json
>   [junit4]   2> 74150 T424 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
>   [junit4]   2> 74155 T398 oascc.SolrZkClient.makePath makePath: /aliases.json
>   [junit4]   2> 74158 T398 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 74164 T423 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
>   [junit4]   2> 74167 T425 oasc.ZkController.publish publishing core=collection1 state=down
>   [junit4]   2> 74168 T425 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
>   [junit4]   2> 74170 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 74170 T425 oasc.ZkController.waitForCoreNodeName look for our core node name
>   [junit4]   2> 74173 T423 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 74174 T423 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"down",
>   [junit4]   2> 	  "base_url":"https://127.0.0.1:64780/sv",
>   [junit4]   2> 	  "core":"collection1",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:64780_sv",
>   [junit4]   2> 	  "shard":null,
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"control_collection",
>   [junit4]   2> 	  "numShards":"1",
>   [junit4]   2> 	  "core_node_name":null}
>   [junit4]   2> 74174 T423 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
>   [junit4]   2> 74174 T423 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
>   [junit4]   2> 74183 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 74185 T422 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>   [junit4]   2> 75196 T425 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
>   [junit4]   2> 75196 T425 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: .\org.apache.solr.cloud.RecoveryZkTest-controljetty-1392655614778\collection1
>   [junit4]   2> 75196 T425 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
>   [junit4]   2> 75198 T425 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>   [junit4]   2> 75198 T425 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
>   [junit4]   2> 75200 T425 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
>   [junit4]   2> 75200 T425 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '.\org.apache.solr.cloud.RecoveryZkTest-controljetty-1392655614778\collection1\'
>   [junit4]   2> 75202 T425 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1392655614778/collection1/lib/classes/' to classloader
>   [junit4]   2> 75203 T425 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1392655614778/collection1/lib/README' to classloader
>   [junit4]   2> 75252 T425 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
>   [junit4]   2> 75265 T425 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>   [junit4]   2> 75269 T425 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>   [junit4]   2> 75273 T425 oass.IndexSchema.readSchema [collection1] Schema name=test
>   [junit4]   2> 75402 T425 oass.IndexSchema.readSchema default search field in schema is text
>   [junit4]   2> 75404 T425 oass.IndexSchema.readSchema unique key field: id
>   [junit4]   2> 75405 T425 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 75410 T425 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 75451 T425 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
>   [junit4]   2> 75452 T425 oasc.SolrCore.<init> [collection1] Opening new SolrCore at .\org.apache.solr.cloud.RecoveryZkTest-controljetty-1392655614778\collection1\, dataDir=.\org.apache.solr.cloud.RecoveryZkTest-1392655614515/control/data\
>   [junit4]   2> 75453 T425 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@76db10
>   [junit4]   2> 75453 T425 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.RecoveryZkTest-1392655614515/control/data\
>   [junit4]   2> 75453 T425 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=.\org.apache.solr.cloud.RecoveryZkTest-1392655614515/control/data\index/
>   [junit4]   2> 75454 T425 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '.\org.apache.solr.cloud.RecoveryZkTest-1392655614515\control\data\index' doesn't exist. Creating new index...
>   [junit4]   2> 75454 T425 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.RecoveryZkTest-1392655614515/control/data\index
>   [junit4]   2> 75454 T425 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=45, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5542235393805511]
>   [junit4]   2> 75455 T425 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
>   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1da363e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3e3a60),segFN=segments_1,generation=1}
>   [junit4]   2> 75455 T425 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 75459 T425 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
>   [junit4]   2> 75459 T425 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
>   [junit4]   2> 75460 T425 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
>   [junit4]   2> 75460 T425 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 75460 T425 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 75460 T425 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>   [junit4]   2> 75461 T425 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 75461 T425 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 75461 T425 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
>   [junit4]   2> 75463 T425 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
>   [junit4]   2> 75463 T425 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
>   [junit4]   2> 75463 T425 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
>   [junit4]   2> 75463 T425 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
>   [junit4]   2> 75464 T425 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
>   [junit4]   2> 75464 T425 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
>   [junit4]   2> 75464 T425 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
>   [junit4]   2> 75465 T425 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
>   [junit4]   2> 75471 T425 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>   [junit4]   2> 75477 T425 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>   [junit4]   2> 75478 T425 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>   [junit4]   2> 75479 T425 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=16, maxMergedSegmentMB=5.0927734375, floorSegmentMB=0.62890625, forceMergeDeletesPctAllowed=14.302540920360697, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
>   [junit4]   2> 75479 T425 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
>   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1da363e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3e3a60),segFN=segments_1,generation=1}
>   [junit4]   2> 75479 T425 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 75480 T425 oass.SolrIndexSearcher.<init> Opening Searcher@2462b[collection1] main
>   [junit4]   2> 75486 T425 oasc.CoreContainer.registerCore registering core: collection1
>   [junit4]   2> 75486 T426 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2462b[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
>   [junit4]   2> 75488 T398 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
>   [junit4]   2> 75489 T398 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>   [junit4]   2> 75489 T429 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:64780/sv collection:control_collection shard:shard1
>   [junit4]   2> 75497 T398 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 75500 T429 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
>   [junit4]   2> 75508 T431 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@67cd3a name:ZooKeeperConnection Watcher:127.0.0.1:64770/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 75508 T398 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 75516 T398 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 75530 T429 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
>   [junit4]   2> 75530 T398 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
>   [junit4]   2> 75825 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 75828 T429 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
>   [junit4]   2> 75828 T429 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
>   [junit4]   2> ASYNC  NEW_CORE C112 name=collection1 org.apache.solr.core.SolrCore@c7c049 url=https://127.0.0.1:64780/sv/collection1 node=127.0.0.1:64780_sv C112_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:64780/sv, core=collection1, node_name=127.0.0.1:64780_sv}
>   [junit4]   2> 75828 T429 C112 P64780 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:64780/sv/collection1/
>   [junit4]   2> 75828 T429 C112 P64780 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
>   [junit4]   2> 75829 T429 C112 P64780 oasc.SyncStrategy.syncToMe https://127.0.0.1:64780/sv/collection1/ has no replicas
>   [junit4]   2> 75829 T429 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:64780/sv/collection1/ shard1
>   [junit4]   2> 75829 T429 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
>   [junit4]   2> 75830 T423 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 75839 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 75842 T431 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>   [junit4]   2> 75842 T422 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>   [junit4]   2> 75858 T423 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 75864 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 75968 T431 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>   [junit4]   2> 75969 T422 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>   [junit4]   2> 76009 T429 oasc.ZkController.register We are https://127.0.0.1:64780/sv/collection1/ and leader is https://127.0.0.1:64780/sv/collection1/
>   [junit4]   2> 76009 T429 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:64780/sv
>   [junit4]   2> 76009 T429 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
>   [junit4]   2> 76009 T429 oasc.ZkController.publish publishing core=collection1 state=active
>   [junit4]   2> 76009 T429 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
>   [junit4]   2> 76012 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 76012 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 76012 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 76013 T429 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 76016 T423 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 76017 T423 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"active",
>   [junit4]   2> 	  "base_url":"https://127.0.0.1:64780/sv",
>   [junit4]   2> 	  "core":"collection1",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:64780_sv",
>   [junit4]   2> 	  "shard":"shard1",
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"control_collection",
>   [junit4]   2> 	  "numShards":"1",
>   [junit4]   2> 	  "core_node_name":"core_node1"}
>   [junit4]   2> 76027 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 76041 T398 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
>   [junit4]   2> 76042 T398 oejs.Server.doStart jetty-8.1.10.v20130312
>   [junit4]   2> 76054 T398 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
>   [junit4]   2> 76058 T398 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:64792
>   [junit4]   2> 76063 T398 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>   [junit4]   2> 76063 T398 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>   [junit4]   2> 76063 T398 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: .\org.apache.solr.cloud.RecoveryZkTest-jetty1-1392655616889
>   [junit4]   2> 76063 T398 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '.\org.apache.solr.cloud.RecoveryZkTest-jetty1-1392655616889\'
>   [junit4]   2> 76105 T398 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.RecoveryZkTest-jetty1-1392655616889\solr.xml
>   [junit4]   2> 76123 T398 oasc.CoreContainer.<init> New CoreContainer 17638025
>   [junit4]   2> 76123 T398 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=.\org.apache.solr.cloud.RecoveryZkTest-jetty1-1392655616889\]
>   [junit4]   2> 76124 T398 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
>   [junit4]   2> 76124 T398 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
>   [junit4]   2> 76124 T398 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>   [junit4]   2> 76124 T398 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>   [junit4]   2> 76125 T398 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>   [junit4]   2> 76125 T398 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>   [junit4]   2> 76125 T398 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>   [junit4]   2> 76126 T398 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>   [junit4]   2> 76127 T398 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>   [junit4]   2> 76143 T398 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>   [junit4]   2> 76144 T398 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>   [junit4]   2> 76144 T398 oasc.CoreContainer.load Host Name: 127.0.0.1
>   [junit4]   2> 76145 T398 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:64770/solr
>   [junit4]   2> 76145 T398 oasc.ZkController.checkChrootPath zkHost includes chroot
>   [junit4]   2> 76147 T431 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>   [junit4]   2> 76148 T422 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>   [junit4]   2> 76150 T398 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 76158 T442 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19b74a name:ZooKeeperConnection Watcher:127.0.0.1:64770 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 76158 T398 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 76170 T398 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 76174 T444 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15fb95d name:ZooKeeperConnection Watcher:127.0.0.1:64770/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 76175 T398 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 76185 T398 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 77190 T398 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64792_sv
>   [junit4]   2> 77193 T398 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:64792_sv
>   [junit4]   2> 77197 T422 oascc.ZkStateReader$3.process Updating live nodes... (2)
>   [junit4]   2> 77198 T431 oascc.ZkStateReader$3.process Updating live nodes... (2)
>   [junit4]   2> 77198 T444 oascc.ZkStateReader$3.process Updating live nodes... (2)
>   [junit4]   2> 77207 T445 oasc.ZkController.publish publishing core=collection1 state=down
>   [junit4]   2> 77207 T445 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
>   [junit4]   2> 77209 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 77209 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 77209 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 77209 T445 oasc.ZkController.waitForCoreNodeName look for our core node name
>   [junit4]   2> 77212 T423 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 77214 T423 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"down",
>   [junit4]   2> 	  "base_url":"https://127.0.0.1:64792/sv",
>   [junit4]   2> 	  "core":"collection1",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:64792_sv",
>   [junit4]   2> 	  "shard":null,
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"collection1",
>   [junit4]   2> 	  "numShards":"1",
>   [junit4]   2> 	  "core_node_name":null}
>   [junit4]   2> 77214 T423 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
>   [junit4]   2> 77214 T423 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
>   [junit4]   2> 77220 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 77324 T444 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>   [junit4]   2> 77325 T431 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>   [junit4]   2> 77325 T422 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>   [junit4]   2> 78210 T445 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
>   [junit4]   2> 78210 T445 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: .\org.apache.solr.cloud.RecoveryZkTest-jetty1-1392655616889\collection1
>   [junit4]   2> 78210 T445 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
>   [junit4]   2> 78211 T445 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>   [junit4]   2> 78212 T445 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
>   [junit4]   2> 78214 T445 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
>   [junit4]   2> 78214 T445 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '.\org.apache.solr.cloud.RecoveryZkTest-jetty1-1392655616889\collection1\'
>   [junit4]   2> 78216 T445 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1392655616889/collection1/lib/classes/' to classloader
>   [junit4]   2> 78217 T445 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1392655616889/collection1/lib/README' to classloader
>   [junit4]   2> 78246 T445 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
>   [junit4]   2> 78266 T445 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>   [junit4]   2> 78270 T445 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>   [junit4]   2> 78278 T445 oass.IndexSchema.readSchema [collection1] Schema name=test
>   [junit4]   2> 78405 T445 oass.IndexSchema.readSchema default search field in schema is text
>   [junit4]   2> 78406 T445 oass.IndexSchema.readSchema unique key field: id
>   [junit4]   2> 78407 T445 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 78424 T445 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 78476 T445 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
>   [junit4]   2> 78477 T445 oasc.SolrCore.<init> [collection1] Opening new SolrCore at .\org.apache.solr.cloud.RecoveryZkTest-jetty1-1392655616889\collection1\, dataDir=.\org.apache.solr.cloud.RecoveryZkTest-1392655614515/jetty1\
>   [junit4]   2> 78477 T445 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@76db10
>   [junit4]   2> 78478 T445 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.RecoveryZkTest-1392655614515/jetty1\
>   [junit4]   2> 78478 T445 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=.\org.apache.solr.cloud.RecoveryZkTest-1392655614515/jetty1\index/
>   [junit4]   2> 78478 T445 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '.\org.apache.solr.cloud.RecoveryZkTest-1392655614515\jetty1\index' doesn't exist. Creating new index...
>   [junit4]   2> 78479 T445 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.RecoveryZkTest-1392655614515/jetty1\index
>   [junit4]   2> 78479 T445 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=45, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5542235393805511]
>   [junit4]   2> 78480 T445 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
>   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@33a841 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1f5f137),segFN=segments_1,generation=1}
>   [junit4]   2> 78480 T445 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 78484 T445 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
>   [junit4]   2> 78484 T445 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
>   [junit4]   2> 78484 T445 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
>   [junit4]   2> 78484 T445 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 78485 T445 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 78485 T445 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>   [junit4]   2> 78485 T445 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 78486 T445 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 78486 T445 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
>   [junit4]   2> 78486 T445 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
>   [junit4]   2> 78487 T445 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
>   [junit4]   2> 78487 T445 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
>   [junit4]   2> 78487 T445 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
>   [junit4]   2> 78487 T445 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
>   [junit4]   2> 78487 T445 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
>   [junit4]   2> 78487 T445 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
>   [junit4]   2> 78488 T445 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
>   [junit4]   2> 78494 T445 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>   [junit4]   2> 78498 T445 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>   [junit4]   2> 78498 T445 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>   [junit4]   2> 78499 T445 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=16, maxMergedSegmentMB=5.0927734375, floorSegmentMB=0.62890625, forceMergeDeletesPctAllowed=14.302540920360697, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
>   [junit4]   2> 78499 T445 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
>   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@33a841 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1f5f137),segFN=segments_1,generation=1}
>   [junit4]   2> 78499 T445 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 78499 T445 oass.SolrIndexSearcher.<init> Opening Searcher@103407a[collection1] main
>   [junit4]   2> 78503 T446 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@103407a[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
>   [junit4]   2> 78507 T445 oasc.CoreContainer.registerCore registering core: collection1
>   [junit4]   2> 78509 T398 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
>   [junit4]   2> 78509 T398 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>   [junit4]   2> 78515 T449 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:64792/sv collection:collection1 shard:shard1
>   [junit4]   2> 78518 T449 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
>   [junit4]   2> 78548 T449 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
>   [junit4]   2> 78554 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 78554 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 78554 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 78560 T449 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
>   [junit4]   2> 78560 T449 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
>   [junit4]   2> ASYNC  NEW_CORE C113 name=collection1 org.apache.solr.core.SolrCore@3c7d0b url=https://127.0.0.1:64792/sv/collection1 node=127.0.0.1:64792_sv C113_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:64792/sv, core=collection1, node_name=127.0.0.1:64792_sv}
>   [junit4]   2> 78560 T449 C113 P64792 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:64792/sv/collection1/
>   [junit4]   2> 78560 T449 C113 P64792 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
>   [junit4]   2> 78560 T449 C113 P64792 oasc.SyncStrategy.syncToMe https://127.0.0.1:64792/sv/collection1/ has no replicas
>   [junit4]   2> 78560 T449 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:64792/sv/collection1/ shard1
>   [junit4]   2> 78560 T449 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
>   [junit4]   2> 78574 T423 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 78628 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 78646 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 78656 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 78760 T431 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>   [junit4]   2> 78761 T422 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>   [junit4]   2> 78762 T444 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>   [junit4]   2> 78802 T449 oasc.ZkController.register We are https://127.0.0.1:64792/sv/collection1/ and leader is https://127.0.0.1:64792/sv/collection1/
>   [junit4]   2> 78802 T449 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:64792/sv
>   [junit4]   2> 78802 T449 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
>   [junit4]   2> 78802 T449 oasc.ZkController.publish publishing core=collection1 state=active
>   [junit4]   2> 78802 T449 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
>   [junit4]   2> 78807 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 78807 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 78807 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 78810 T449 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 78813 T423 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 78816 T423 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"active",
>   [junit4]   2> 	  "base_url":"https://127.0.0.1:64792/sv",
>   [junit4]   2> 	  "core":"collection1",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:64792_sv",
>   [junit4]   2> 	  "shard":"shard1",
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"collection1",
>   [junit4]   2> 	  "numShards":"1",
>   [junit4]   2> 	  "core_node_name":"core_node1"}
>   [junit4]   2> 78823 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 78939 T444 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>   [junit4]   2> 78940 T431 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>   [junit4]   2> 78940 T422 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>   [junit4]   2> 79075 T398 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
>   [junit4]   2> 79076 T398 oejs.Server.doStart jetty-8.1.10.v20130312
>   [junit4]   2> 79088 T398 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
>   [junit4]   2> 79092 T398 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:64801
>   [junit4]   2> 79096 T398 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>   [junit4]   2> 79096 T398 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>   [junit4]   2> 79097 T398 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.RecoveryZkTest-jetty2-1392655619874
>   [junit4]   2> 79097 T398 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.RecoveryZkTest-jetty2-1392655619874\'
>   [junit4]   2> 79125 T398 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.RecoveryZkTest-jetty2-1392655619874\solr.xml
>   [junit4]   2> 79148 T398 oasc.CoreContainer.<init> New CoreContainer 22453648
>   [junit4]   2> 79148 T398 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.RecoveryZkTest-jetty2-1392655619874\]
>   [junit4]   2> 79150 T398 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
>   [junit4]   2> 79150 T398 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
>   [junit4]   2> 79150 T398 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>   [junit4]   2> 79150 T398 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>   [junit4]   2> 79150 T398 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>   [junit4]   2> 79150 T398 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>   [junit4]   2> 79150 T398 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>   [junit4]   2> 79150 T398 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>   [junit4]   2> 79150 T398 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>   [junit4]   2> 79160 T398 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>   [junit4]   2> 79160 T398 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>   [junit4]   2> 79161 T398 oasc.CoreContainer.load Host Name: 127.0.0.1
>   [junit4]   2> 79161 T398 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:64770/solr
>   [junit4]   2> 79163 T398 oasc.ZkController.checkChrootPath zkHost includes chroot
>   [junit4]   2> 79164 T398 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 79170 T460 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f748b name:ZooKeeperConnection Watcher:127.0.0.1:64770 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 79171 T398 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 79175 T400 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14440be423e0008, likely client has closed socket
>   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
>   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
>   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
>   [junit4]   2> 
>   [junit4]   2> 79177 T398 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 79180 T462 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15bfbf0 name:ZooKeeperConnection Watcher:127.0.0.1:64770/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 79181 T398 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 79190 T398 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 80195 T398 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64801_sv
>   [junit4]   2> 80197 T398 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:64801_sv
>   [junit4]   2> 80204 T444 oascc.ZkStateReader$3.process Updating live nodes... (3)
>   [junit4]   2> 80206 T431 oascc.ZkStateReader$3.process Updating live nodes... (3)
>   [junit4]   2> 80206 T422 oascc.ZkStateReader$3.process Updating live nodes... (3)
>   [junit4]   2> 80208 T462 oascc.ZkStateReader$3.process Updating live nodes... (3)
>   [junit4]   2> 80218 T463 oasc.ZkController.publish publishing core=collection1 state=down
>   [junit4]   2> 80218 T463 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
>   [junit4]   2> 80221 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 80221 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 80221 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 80221 T463 oasc.ZkController.waitForCoreNodeName look for our core node name
>   [junit4]   2> 80224 T423 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 80225 T423 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"down",
>   [junit4]   2> 	  "base_url":"https://127.0.0.1:64801/sv",
>   [junit4]   2> 	  "core":"collection1",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:64801_sv",
>   [junit4]   2> 	  "shard":null,
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"collection1",
>   [junit4]   2> 	  "numShards":"1",
>   [junit4]   2> 	  "core_node_name":null}
>   [junit4]   2> 80226 T423 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
>   [junit4]   2> 80226 T423 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
>   [junit4]   2> 80231 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 80360 T431 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>   [junit4]   2> 80360 T462 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>   [junit4]   2> 80360 T444 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>   [junit4]   2> 80360 T422 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>   [junit4]   2> 81222 T463 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
>   [junit4]   2> 81222 T463 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.RecoveryZkTest-jetty2-1392655619874\collection1
>   [junit4]   2> 81222 T463 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
>   [junit4]   2> 81224 T463 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>   [junit4]   2> 81224 T463 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
>   [junit4]   2> 81226 T463 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
>   [junit4]   2> 81226 T463 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.RecoveryZkTest-jetty2-1392655619874\collection1\'
>   [junit4]   2> 81229 T463 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1392655619874/collection1/lib/classes/' to classloader
>   [junit4]   2> 81231 T463 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1392655619874/collection1/lib/README' to classloader
>   [junit4]   2> 81275 T463 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
>   [junit4]   2> 81292 T463 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>   [junit4]   2> 81297 T463 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>   [junit4]   2> 81303 T463 oass.IndexSchema.readSchema [collection1] Schema name=test
>   [junit4]   2> 81433 T463 oass.IndexSchema.readSchema default search field in schema is text
>   [junit4]   2> 81434 T463 oass.IndexSchema.readSchema unique key field: id
>   [junit4]   2> 81435 T463 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 81440 T463 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 81496 T463 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
>   [junit4]   2> 81497 T463 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.RecoveryZkTest-jetty2-1392655619874\collection1\, dataDir=.\org.apache.solr.cloud.RecoveryZkTest-1392655614515/jetty2\
>   [junit4]   2> 81497 T463 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@76db10
>   [junit4]   2> 81497 T463 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.RecoveryZkTest-1392655614515/jetty2\
>   [junit4]   2> 81497 T463 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=.\org.apache.solr.cloud.RecoveryZkTest-1392655614515/jetty2\index/
>   [junit4]   2> 81497 T463 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '.\org.apache.solr.cloud.RecoveryZkTest-1392655614515\jetty2\index' doesn't exist. Creating new index...
>   [junit4]   2> 81499 T463 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.RecoveryZkTest-1392655614515/jetty2\index
>   [junit4]   2> 81500 T463 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=45, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5542235393805511]
>   [junit4]   2> 81500 T463 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
>   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1239d05 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@13a8dfb),segFN=segments_1,generation=1}
>   [junit4]   2> 81500 T463 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 81504 T463 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
>   [junit4]   2> 81505 T463 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
>   [junit4]   2> 81505 T463 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
>   [junit4]   2> 81505 T463 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 81505 T463 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 81506 T463 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>   [junit4]   2> 81506 T463 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 81506 T463 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 81506 T463 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
>   [junit4]   2> 81507 T463 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
>   [junit4]   2> 81507 T463 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
>   [junit4]   2> 81507 T463 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
>   [junit4]   2> 81508 T463 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
>   [junit4]   2> 81508 T463 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
>   [junit4]   2> 81508 T463 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
>   [junit4]   2> 81509 T463 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
>   [junit4]   2> 81509 T463 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
>   [junit4]   2> 81515 T463 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>   [junit4]   2> 81519 T463 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>   [junit4]   2> 81519 T463 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>   [junit4]   2> 81520 T463 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=16, maxMergedSegmentMB=5.0927734375, floorSegmentMB=0.62890625, forceMergeDeletesPctAllowed=14.302540920360697, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
>   [junit4]   2> 81521 T463 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
>   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1239d05 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@13a8dfb),segFN=segments_1,generation=1}
>   [junit4]   2> 81521 T463 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 81521 T463 oass.SolrIndexSearcher.<init> Opening Searcher@e14a16[collection1] main
>   [junit4]   2> 81526 T464 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@e14a16[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
>   [junit4]   2> 81529 T463 oasc.CoreContainer.registerCore registering core: collection1
>   [junit4]   2> 81529 T398 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
>   [junit4]   2> 81529 T398 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>   [junit4]   2> 81529 T467 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:64801/sv collection:collection1 shard:shard1
>   [junit4]   2> 81536 T398 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 81539 T467 oasc.ZkController.register We are https://127.0.0.1:64801/sv/collection1/ and leader is https://127.0.0.1:64792/sv/collection1/
>   [junit4]   2> 81539 T467 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:64801/sv
>   [junit4]   2> 81539 T467 oasc.ZkController.checkRecovery Core needs to recover:collection1
>   [junit4]   2> 81541 T467 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
>   [junit4]   2> ASYNC  NEW_CORE C114 name=collection1 org.apache.solr.core.SolrCore@18447d2 url=https://127.0.0.1:64801/sv/collection1 node=127.0.0.1:64801_sv C114_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:64801/sv, core=collection1, node_name=127.0.0.1:64801_sv}
>   [junit4]   2> 81551 T470 C114 P64801 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
>   [junit4]   2> 81552 T470 C114 P64801 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
>   [junit4]   2> 81552 T470 C114 P64801 oasc.ZkController.publish publishing core=collection1 state=recovering
>   [junit4]   2> 81552 T470 C114 P64801 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
>   [junit4]   2> 81564 T467 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> ASYNC  NEW_CORE C115 name=collection1 org.apache.solr.core.SolrCore@c7c049 url=https://127.0.0.1:64780/sv/collection1 node=127.0.0.1:64780_sv C115_STATE=coll:control_collection core:collection1 props:{state=active, base_url=https://127.0.0.1:64780/sv, core=collection1, node_name=127.0.0.1:64780_sv, leader=true}
>   [junit4]   2> 81679 T414 C115 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-0 (1460305262576402432)]} 0 23
>   [junit4]   2> 81692 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 81692 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 81692 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 81760 T398 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 64801
>   [junit4]   2> 81760 T398 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=22453648
>   [junit4]   2> 81760 T398 oasc.RecoveryStrategy.close WARN Stopping recovery for zkNodeName=core_node2core=collection1
>   [junit4]   2> 81766 T423 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 81827 T423 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"recovering",
>   [junit4]   2> 	  "base_url":"https://127.0.0.1:64801/sv",
>   [junit4]   2> 	  "core":"collection1",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:64801_sv",
>   [junit4]   2> 	  "shard":"shard1",
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"collection1",
>   [junit4]   2> 	  "numShards":"1",
>   [junit4]   2> 	  "core_node_name":"core_node2"}
>   [junit4]   2> 81865 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 81867 T418 C115 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-0 (1460305262776680448)]} 0 4
>   [junit4]   2> 81876 T444 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>   [junit4]   2> 81877 T431 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>   [junit4]   2> 81878 T462 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>   [junit4]   2> 81878 T422 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>   [junit4]   2> ASYNC  NEW_CORE C116 name=collection1 org.apache.solr.core.SolrCore@3c7d0b url=https://127.0.0.1:64792/sv/collection1 node=127.0.0.1:64792_sv C116_STATE=coll:collection1 core:collection1 props:{state=active, base_url=https://127.0.0.1:64792/sv, core=collection1, node_name=127.0.0.1:64792_sv, leader=true}
>   [junit4]   2> 81889 T435 C116 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-0 (1460305262786117632)]} 0 28
>   [junit4]   2> 81892 T437 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true
>   [junit4]   2> 81932 T437 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 0 seconds.
>   [junit4]   2> 81933 T437 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:64801_sv&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node2&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=41 
>   [junit4]   2> 81968 T413 C115 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-1 (1460305262885732352)]} 0 1
>   [junit4]   2> ASYNC  NEW_CORE C117 name=collection1 org.apache.solr.core.SolrCore@18447d2 url=https://127.0.0.1:64801/sv/collection1 node=127.0.0.1:64801_sv C117_STATE=coll:collection1 core:collection1 props:{state=recovering, base_url=https://127.0.0.1:64801/sv, core=collection1, node_name=127.0.0.1:64801_sv}
>   [junit4]   2> 82137 T458 C117 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[2-1 (1460305262902509568)]} 0 10
>   [junit4]   2> 82139 T438 C116 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-1 (1460305262902509568)]} 0 155
>   [junit4]   2> 82146 T455 C117 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[1-0 (1460305262863712256)]} 0 4
>   [junit4]   2> 82151 T437 C116 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-0 (1460305262863712256)]} 0 212
>   [junit4]   2> 82173 T415 C115 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-2 (1460305263099641856)]} 0 2
>   [junit4]   2> 82182 T417 C115 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-1 (1460305263109079040)]} 0 2
>   [junit4]   2> 82276 T457 C117 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[2-2 (1460305263130050560)]} 0 5
>   [junit4]   2> 82279 T435 C116 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-2 (1460305263130050560)]} 0 78
>   [junit4]   2> 82297 T453 C117 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[1-1 (1460305263200305152)]} 0 0
>   [junit4]   2> 82308 T438 C116 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-1 (1460305263200305152)]} 0 47
>   [junit4]   2> 82338 T414 C115 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-3 (1460305263272656896)]} 0 5
>   [junit4]   2> 82347 T414 C115 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-2 (1460305263283142656)]} 0 1
>   [junit4]   2> 82427 T455 C117 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[2-3 (1460305263310405632)]} 0 1
>   [junit4]   2> 82435 T435 C116 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-3 (1460305263310405632)]} 0 63
>   [junit4]   2> 82442 T453 C117 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[1-2 (1460305263327182848)]} 0 1
>   [junit4]   2> 82445 T439 C116 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-2 (1460305263327182848)]} 0 56
>   [junit4]   2> 82468 T417 C115 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-3 (1460305263408971776)]} 0 1
>   [junit4]   2> 82518 T416 C115 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2&CONTROL=TRUE} {delete=[2-1 (-1460305263461400576)]} 0 1
>   [junit4]   2> 82523 T458 C117 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[1-3 (1460305263424700416)]} 0 7
>   [junit4]   2> 82529 T437 C116 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-3 (1460305263424700416)]} 0 49
>   [junit4]   2> 82561 T455 C117 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {delete=[2-1 (-1460305263484469248)]} 0 1
>   [junit4]   2> 82566 T439 C116 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {delete=[2-1 (-1460305263484469248)]} 0 28
>   [junit4]   2> 82602 T415 C115 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-4 (1460305263548432384)]} 0 2
>   [junit4]   2> ASYNC  NEW_CORE C118 name=collection1 org.apache.solr.core.SolrCore@c7c049 url=https://127.0.0.1:64780/sv/collection1 node=127.0.0.1:64780_sv C118_STATE=coll:control_collection core:collection1 props:{state=active, base_url=https://127.0.0.1:64780/sv, core=collection1, node_name=127.0.0.1:64780_sv, leader=true}
>   [junit4]   2> 82658 T418 C118 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2&CONTROL=TRUE} {delete=[1-1 (-1460305263609249792)]} 0 1
>   [junit4]   2> ASYNC  NEW_CORE C119 name=collection1 org.apache.solr.core.SolrCore@18447d2 url=https://127.0.0.1:64801/sv/collection1 node=127.0.0.1:64801_sv C119_STATE=coll:collection1 core:collection1 props:{state=recovering, base_url=https://127.0.0.1:64801/sv, core=collection1, node_name=127.0.0.1:64801_sv}
>   [junit4]   2> 82985 T458 C119 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {delete=[1-1 (-1460305263633367040)]} 0 1
>   [junit4]   2> ASYNC  NEW_CORE C120 name=collection1 org.apache.solr.core.SolrCore@3c7d0b url=https://127.0.0.1:64792/sv/collection1 node=127.0.0.1:64792_sv C120_STATE=coll:collection1 core:collection1 props:{state=active, base_url=https://127.0.0.1:64792/sv, core=collection1, node_name=127.0.0.1:64792_sv, leader=true}
>   [junit4]   2> 82987 T435 C120 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {delete=[1-1 (-1460305263633367040)]} 0 307
>   [junit4]   2> 82994 T456 C119 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[2-4 (1460305263651192832)]} 0 0
>   [junit4]   2> 82997 T439 C120 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-4 (1460305263651192832)]} 0 299
>   [junit4]   2> 83011 T413 C118 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-4 (1460305263979397120)]} 0 2
>   [junit4]   2> 83057 T457 C119 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[1-4 (1460305263994077184)]} 0 2
>   [junit4]   2> 83064 T436 C120 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-4 (1460305263994077184)]} 0 39
>   [junit4]   2> 83070 T414 C118 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2&CONTROL=TRUE} {delete=[2-2 (-1460305264041263104)]} 0 3
>   [junit4]   2> 83157 T458 C119 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {delete=[2-2 (-1460305264059088896)]} 0 1
>   [junit4]   2> 83159 T435 C120 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {delete=[2-2 (-1460305264059088896)]} 0 72
>   [junit4]   2> 83162 T417 C118 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2&CONTROL=TRUE} {delete=[1-2 (-1460305264137732096)]} 0 1
>   [junit4]   2> 83183 T418 C118 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-5 (1460305264159752192)]} 0 0
>   [junit4]   2> 83247 T456 C119 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {delete=[1-2 (-1460305264171286528)]} 0 1
>   [junit4]   2> 83249 T440 C120 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {delete=[1-2 (-1460305264171286528)]} 0 56
>   [junit4]   2> 83265 T453 C119 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[2-5 (1460305264225812480)]} 0 0
>   [junit4]   2> 83268 T435 C120 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-5 (1460305264225812480)]} 0 22
>   [junit4]   2> 83271 T416 C118 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-5 (1460305264245735424)]} 0 6
>   [junit4]   2> 83300 T454 C119 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[1-5 (1460305264268804096)]} 0 2
>   [junit4]   2> 83303 T438 C120 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-5 (1460305264268804096)]} 0 16
>   [junit4]   2> 83310 T414 C118 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-6 (1460305264292921344)]} 0 1
>   [junit4]   2> 83327 T415 C118 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-6 (1460305264310747136)]} 0 1
>   [junit4]   2> 83494 T455 C119 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[2-6 (1460305264312844288)]} 0 5
>   [junit4]   2> 83497 T437 C120 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-6 (1460305264312844288)]} 0 168
>   [junit4]   2> 83520 T453 C119 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[1-6 (1460305264485859328)]} 0 1
>   [junit4]   2> 83522 T435 C120 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-6 (1460305264485859328)]} 0 28
>   [junit4]   2> 83546 T416 C118 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-7 (1460305264540385280)]} 0 1
>   [junit4]   2> 83572 T414 C118 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-7 (1460305264567648256)]} 0 1
>   [junit4]   2> 83580 T458 C119 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[2-7 (1460305264550871040)]} 0 1
>   [junit4]   2> 83582 T439 C120 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-7 (1460305264550871040)]} 0 26
>   [junit4]   2> 83606 T457 C119 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[1-7 (1460305264587571200)]} 0 1
>   [junit4]   2> 83610 T439 C120 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-7 (1460305264587571200)]} 0 19
>   [junit4]   2> 83650 T417 C118 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-8 (1460305264649437184)]} 0 1
>   [junit4]   2> 83683 T418 C118 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-8 (1460305264682991616)]} 0 2
>   [junit4]   2> 83690 T458 C119 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[2-8 (1460305264664117248)]} 0 1
>   [junit4]   2> 83692 T440 C120 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-8 (1460305264664117248)]} 0 28
>   [junit4]   2> 83719 T455 C119 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[1-8 (1460305264701865984)]} 0 9
>   [junit4]   2> 83720 T439 C120 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-8 (1460305264701865984)]} 0 21
>   [junit4]   2> 83804 T414 C118 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-9 (1460305264809869312)]} 0 1
>   [junit4]   2> 83845 T458 C119 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[2-9 (1460305264824549376)]} 0 0
>   [junit4]   2> 83848 T437 C120 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-9 (1460305264824549376)]} 0 32
>   [junit4]   2> 83861 T415 C118 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-9 (1460305264869638144)]} 0 1
>   [junit4]   2> ASYNC  NEW_CORE C121 name=collection1 org.apache.solr.core.SolrCore@18447d2 url=https://127.0.0.1:64801/sv/collection1 node=127.0.0.1:64801_sv C121_STATE=coll:collection1 core:collection1 props:{state=recovering, base_url=https://127.0.0.1:64801/sv, core=collection1, node_name=127.0.0.1:64801_sv}
>   [junit4]   2> 83897 T458 C121 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[1-9 (1460305264897949696)]} 0 0
>   [junit4]   2> ASYNC  NEW_CORE C122 name=collection1 org.apache.solr.core.SolrCore@3c7d0b url=https://127.0.0.1:64792/sv/collection1 node=127.0.0.1:64792_sv C122_STATE=coll:collection1 core:collection1 props:{state=active, base_url=https://127.0.0.1:64792/sv, core=collection1, node_name=127.0.0.1:64792_sv, leader=true}
>   [junit4]   2> 83899 T440 C122 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-9 (1460305264897949696)]} 0 12
>   [junit4]   2> ASYNC  NEW_CORE C123 name=colle
> 
> [...truncated too long message...]
> 
> [junit4]   2> 	        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
>   [junit4]   2> 	        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
>   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
>   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
>   [junit4]   2> 	  15) Thread[id=398, name=TEST-RecoveryZkTest.testDistribSearch-seed#[41014895D1172F7D], state=RUNNABLE, group=TGRP-RecoveryZkTest]
>   [junit4]   2> 	        at java.lang.Throwable.fillInStackTrace(Native Method)
>   [junit4]   2> 	        at java.lang.Throwable.fillInStackTrace(Throwable.java:783)
>   [junit4]   2> 	        at java.lang.Throwable.<init>(Throwable.java:250)
>   [junit4]   2> 	        at java.lang.Exception.<init>(Exception.java:54)
>   [junit4]   2> 	        at java.lang.InterruptedException.<init>(InterruptedException.java:57)
>   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2067)
>   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1465)
>   [junit4]   2> 	        at org.apache.solr.common.util.ExecutorUtil.shutdownAndAwaitTermination(ExecutorUtil.java:54)
>   [junit4]   2> 	        at org.apache.solr.update.UpdateShardHandler.close(UpdateShardHandler.java:75)
>   [junit4]   2> 	        at org.apache.solr.core.CoreContainer.shutdown(CoreContainer.java:402)
>   [junit4]   2> 	        at org.apache.solr.servlet.SolrDispatchFilter.destroy(SolrDispatchFilter.java:210)
>   [junit4]   2> 	        at org.apache.solr.cloud.ChaosMonkey.stopJettySolrRunner(ChaosMonkey.java:224)
>   [junit4]   2> 	        at org.apache.solr.cloud.ChaosMonkey.stop(ChaosMonkey.java:563)
>   [junit4]   2> 	        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.destroyServers(AbstractFullDistribZkTestBase.java:1586)
>   [junit4]   2> 	        at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:871)
>   [junit4]   2> 	        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   [junit4]   2> 	        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   [junit4]   2> 	        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   [junit4]   2> 	        at java.lang.reflect.Method.invoke(Method.java:483)
>   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
>   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
>   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
>   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
>   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
>   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>   [junit4]   2> 	        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>   [junit4]   2> 	        at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
>   [junit4]   2> 	        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>   [junit4]   2> 	        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>   [junit4]   2> 	        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
>   [junit4]   2> 	        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
>   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
>   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
>   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
>   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
>   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
>   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
>   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>   [junit4]   2> 	        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>   [junit4]   2> 	        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>   [junit4]   2> 	        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
>   [junit4]   2> 	        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>   [junit4]   2> 	        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
>   [junit4]   2> 	        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
>   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
>   [junit4]   2> NOTE: test params are: codec=Asserting, sim=DefaultSimilarity, locale=es_PA, timezone=America/Los_Angeles
>   [junit4]   2> NOTE: Windows 7 6.1 x86/Oracle Corporation 1.8.0 (32-bit)/cpus=2,threads=21,free=75863024,total=123240448
>   [junit4]   2> NOTE: All tests run in this JVM: [LukeRequestHandlerTest, TestSolrDeletionPolicy2, TestCSVResponseWriter, TestJmxMonitoredMap, DistributedSpellCheckComponentTest, DistributedQueryComponentCustomSortTest, SuggesterFSTTest, ResourceLoaderTest, DeleteShardTest, FieldAnalysisRequestHandlerTest, SimplePostToolTest, ConnectionManagerTest, FileBasedSpellCheckerTest, UpdateParamsTest, StandardRequestHandlerTest, SpellCheckComponentTest, TestSolrCoreProperties, HighlighterTest, RecoveryZkTest]
>   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=RecoveryZkTest -Dtests.seed=41014895D1172F7D -Dtests.slow=true -Dtests.locale=es_PA -Dtests.timezone=America/Los_Angeles -Dtests.file.encoding=Cp1252
>   [junit4] ERROR   0.00s | RecoveryZkTest (suite) <<<
>   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
>   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([41014895D1172F7D]:0)
>   [junit4] Completed in 7268.83s, 1 test, 2 errors <<< FAILURES!
> 
> [...truncated 1066 lines...]
> BUILD FAILED
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:453: The following error occurred while executing this line:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:433: The following error occurred while executing this line:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:39: The following error occurred while executing this line:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-targets.xml:37: The following error occurred while executing this line:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build.xml:189: The following error occurred while executing this line:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\common-build.xml:491: The following error occurred while executing this line:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:1268: The following error occurred while executing this line:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:901: There were test failures: 372 suites, 1604 tests, 1 suite-level error, 1 error, 1433 ignored
> 
> Total time: 154 minutes 45 seconds
> Build step 'Invoke Ant' marked build as failure
> Description set: Java: 32bit/jdk1.8.0-fcs-b129 -server -XX:+UseSerialGC
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure
> Sending email for trigger: Failure
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[JENKINS] Lucene-Solr-trunk-Windows (32bit/jdk1.8.0-fcs-b129) - Build # 3790 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/3790/
Java: 32bit/jdk1.8.0-fcs-b129 -server -XX:+UseSerialGC

2 tests failed.
REGRESSION:  org.apache.solr.cloud.RecoveryZkTest.testDistribSearch

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([41014895D1172F7D]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([41014895D1172F7D]:0)




Build Log:
[...truncated 10038 lines...]
   [junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
   [junit4]   2> 73152 T398 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 73153 T398 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /sv/
   [junit4]   2> 73156 T398 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\solrtest-RecoveryZkTest-1392655614516
   [junit4]   2> 73158 T398 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 73159 T399 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 73263 T398 oasc.ZkTestServer.run start zk server on port:64770
   [junit4]   2> 73265 T398 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 73273 T405 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1de7949 name:ZooKeeperConnection Watcher:127.0.0.1:64770 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 73273 T398 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 73273 T398 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 73284 T398 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 73287 T407 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@184e80 name:ZooKeeperConnection Watcher:127.0.0.1:64770/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 73288 T398 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 73288 T398 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 73294 T398 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 73300 T398 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 73303 T398 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 73308 T398 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 73308 T398 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 73317 T398 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 73317 T398 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 73325 T398 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 73325 T398 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 73331 T398 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 73331 T398 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 73337 T398 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 73338 T398 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 73343 T398 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 73343 T398 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 73358 T398 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 73358 T398 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 73363 T398 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 73363 T398 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 73369 T398 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 73370 T398 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 73375 T398 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 73376 T398 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 73381 T398 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 73381 T398 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 73391 T398 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 73414 T409 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13b72d0 name:ZooKeeperConnection Watcher:127.0.0.1:64770/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 73414 T398 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 73419 T400 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14440be423e0002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 73942 T398 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 73958 T398 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 73969 T398 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:64780
   [junit4]   2> 73974 T398 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 73975 T398 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 73975 T398 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: .\org.apache.solr.cloud.RecoveryZkTest-controljetty-1392655614778
   [junit4]   2> 73975 T398 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '.\org.apache.solr.cloud.RecoveryZkTest-controljetty-1392655614778\'
   [junit4]   2> 74013 T398 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.RecoveryZkTest-controljetty-1392655614778\solr.xml
   [junit4]   2> 74030 T398 oasc.CoreContainer.<init> New CoreContainer 14266042
   [junit4]   2> 74031 T398 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=.\org.apache.solr.cloud.RecoveryZkTest-controljetty-1392655614778\]
   [junit4]   2> 74032 T398 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 74033 T398 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 74033 T398 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 74033 T398 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 74033 T398 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 74033 T398 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 74034 T398 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 74034 T398 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 74034 T398 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 74052 T398 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 74052 T398 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 74052 T398 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 74052 T398 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:64770/solr
   [junit4]   2> 74052 T398 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 74056 T398 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 74068 T420 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7393ab name:ZooKeeperConnection Watcher:127.0.0.1:64770 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 74070 T398 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 74080 T398 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 74086 T422 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b2269b name:ZooKeeperConnection Watcher:127.0.0.1:64770/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 74086 T398 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 74089 T398 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 74096 T398 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 74104 T398 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 74109 T398 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64780_sv
   [junit4]   2> 74111 T398 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:64780_sv
   [junit4]   2> 74117 T398 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 74121 T398 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 74130 T398 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:64780_sv
   [junit4]   2> 74132 T398 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 74135 T398 oasc.Overseer.start Overseer (id=91269078353575940-127.0.0.1:64780_sv-n_0000000000) starting
   [junit4]   2> 74141 T398 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 74148 T424 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 74150 T398 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 74150 T424 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 74155 T398 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 74158 T398 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 74164 T423 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 74167 T425 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 74168 T425 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 74170 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 74170 T425 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 74173 T423 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 74174 T423 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:64780/sv",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:64780_sv",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 74174 T423 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 74174 T423 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 74183 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 74185 T422 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 75196 T425 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 75196 T425 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: .\org.apache.solr.cloud.RecoveryZkTest-controljetty-1392655614778\collection1
   [junit4]   2> 75196 T425 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 75198 T425 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 75198 T425 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 75200 T425 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 75200 T425 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '.\org.apache.solr.cloud.RecoveryZkTest-controljetty-1392655614778\collection1\'
   [junit4]   2> 75202 T425 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1392655614778/collection1/lib/classes/' to classloader
   [junit4]   2> 75203 T425 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1392655614778/collection1/lib/README' to classloader
   [junit4]   2> 75252 T425 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 75265 T425 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 75269 T425 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 75273 T425 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 75402 T425 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 75404 T425 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 75405 T425 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 75410 T425 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 75451 T425 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 75452 T425 oasc.SolrCore.<init> [collection1] Opening new SolrCore at .\org.apache.solr.cloud.RecoveryZkTest-controljetty-1392655614778\collection1\, dataDir=.\org.apache.solr.cloud.RecoveryZkTest-1392655614515/control/data\
   [junit4]   2> 75453 T425 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@76db10
   [junit4]   2> 75453 T425 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.RecoveryZkTest-1392655614515/control/data\
   [junit4]   2> 75453 T425 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=.\org.apache.solr.cloud.RecoveryZkTest-1392655614515/control/data\index/
   [junit4]   2> 75454 T425 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '.\org.apache.solr.cloud.RecoveryZkTest-1392655614515\control\data\index' doesn't exist. Creating new index...
   [junit4]   2> 75454 T425 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.RecoveryZkTest-1392655614515/control/data\index
   [junit4]   2> 75454 T425 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=45, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5542235393805511]
   [junit4]   2> 75455 T425 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1da363e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3e3a60),segFN=segments_1,generation=1}
   [junit4]   2> 75455 T425 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 75459 T425 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 75459 T425 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 75460 T425 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 75460 T425 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 75460 T425 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 75460 T425 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 75461 T425 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 75461 T425 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 75461 T425 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 75463 T425 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 75463 T425 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 75463 T425 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 75463 T425 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 75464 T425 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 75464 T425 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 75464 T425 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 75465 T425 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 75471 T425 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 75477 T425 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 75478 T425 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 75479 T425 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=16, maxMergedSegmentMB=5.0927734375, floorSegmentMB=0.62890625, forceMergeDeletesPctAllowed=14.302540920360697, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 75479 T425 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1da363e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3e3a60),segFN=segments_1,generation=1}
   [junit4]   2> 75479 T425 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 75480 T425 oass.SolrIndexSearcher.<init> Opening Searcher@2462b[collection1] main
   [junit4]   2> 75486 T425 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 75486 T426 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2462b[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 75488 T398 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 75489 T398 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 75489 T429 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:64780/sv collection:control_collection shard:shard1
   [junit4]   2> 75497 T398 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 75500 T429 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 75508 T431 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@67cd3a name:ZooKeeperConnection Watcher:127.0.0.1:64770/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 75508 T398 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 75516 T398 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 75530 T429 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 75530 T398 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 75825 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 75828 T429 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 75828 T429 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C112 name=collection1 org.apache.solr.core.SolrCore@c7c049 url=https://127.0.0.1:64780/sv/collection1 node=127.0.0.1:64780_sv C112_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:64780/sv, core=collection1, node_name=127.0.0.1:64780_sv}
   [junit4]   2> 75828 T429 C112 P64780 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:64780/sv/collection1/
   [junit4]   2> 75828 T429 C112 P64780 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 75829 T429 C112 P64780 oasc.SyncStrategy.syncToMe https://127.0.0.1:64780/sv/collection1/ has no replicas
   [junit4]   2> 75829 T429 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:64780/sv/collection1/ shard1
   [junit4]   2> 75829 T429 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 75830 T423 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 75839 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 75842 T431 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 75842 T422 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 75858 T423 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 75864 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 75968 T431 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 75969 T422 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 76009 T429 oasc.ZkController.register We are https://127.0.0.1:64780/sv/collection1/ and leader is https://127.0.0.1:64780/sv/collection1/
   [junit4]   2> 76009 T429 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:64780/sv
   [junit4]   2> 76009 T429 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 76009 T429 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 76009 T429 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 76012 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 76012 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 76012 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 76013 T429 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 76016 T423 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 76017 T423 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:64780/sv",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:64780_sv",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 76027 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 76041 T398 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 76042 T398 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 76054 T398 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 76058 T398 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:64792
   [junit4]   2> 76063 T398 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 76063 T398 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 76063 T398 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: .\org.apache.solr.cloud.RecoveryZkTest-jetty1-1392655616889
   [junit4]   2> 76063 T398 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '.\org.apache.solr.cloud.RecoveryZkTest-jetty1-1392655616889\'
   [junit4]   2> 76105 T398 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.RecoveryZkTest-jetty1-1392655616889\solr.xml
   [junit4]   2> 76123 T398 oasc.CoreContainer.<init> New CoreContainer 17638025
   [junit4]   2> 76123 T398 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=.\org.apache.solr.cloud.RecoveryZkTest-jetty1-1392655616889\]
   [junit4]   2> 76124 T398 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 76124 T398 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 76124 T398 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 76124 T398 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 76125 T398 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 76125 T398 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 76125 T398 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 76126 T398 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 76127 T398 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 76143 T398 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 76144 T398 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 76144 T398 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 76145 T398 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:64770/solr
   [junit4]   2> 76145 T398 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 76147 T431 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 76148 T422 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 76150 T398 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 76158 T442 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19b74a name:ZooKeeperConnection Watcher:127.0.0.1:64770 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 76158 T398 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 76170 T398 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 76174 T444 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15fb95d name:ZooKeeperConnection Watcher:127.0.0.1:64770/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 76175 T398 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 76185 T398 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 77190 T398 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64792_sv
   [junit4]   2> 77193 T398 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:64792_sv
   [junit4]   2> 77197 T422 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 77198 T431 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 77198 T444 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 77207 T445 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 77207 T445 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 77209 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 77209 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 77209 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 77209 T445 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 77212 T423 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 77214 T423 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:64792/sv",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:64792_sv",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 77214 T423 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
   [junit4]   2> 77214 T423 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 77220 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 77324 T444 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 77325 T431 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 77325 T422 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 78210 T445 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 78210 T445 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: .\org.apache.solr.cloud.RecoveryZkTest-jetty1-1392655616889\collection1
   [junit4]   2> 78210 T445 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 78211 T445 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 78212 T445 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 78214 T445 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 78214 T445 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '.\org.apache.solr.cloud.RecoveryZkTest-jetty1-1392655616889\collection1\'
   [junit4]   2> 78216 T445 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1392655616889/collection1/lib/classes/' to classloader
   [junit4]   2> 78217 T445 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1392655616889/collection1/lib/README' to classloader
   [junit4]   2> 78246 T445 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 78266 T445 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 78270 T445 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 78278 T445 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 78405 T445 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 78406 T445 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 78407 T445 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 78424 T445 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 78476 T445 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 78477 T445 oasc.SolrCore.<init> [collection1] Opening new SolrCore at .\org.apache.solr.cloud.RecoveryZkTest-jetty1-1392655616889\collection1\, dataDir=.\org.apache.solr.cloud.RecoveryZkTest-1392655614515/jetty1\
   [junit4]   2> 78477 T445 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@76db10
   [junit4]   2> 78478 T445 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.RecoveryZkTest-1392655614515/jetty1\
   [junit4]   2> 78478 T445 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=.\org.apache.solr.cloud.RecoveryZkTest-1392655614515/jetty1\index/
   [junit4]   2> 78478 T445 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '.\org.apache.solr.cloud.RecoveryZkTest-1392655614515\jetty1\index' doesn't exist. Creating new index...
   [junit4]   2> 78479 T445 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.RecoveryZkTest-1392655614515/jetty1\index
   [junit4]   2> 78479 T445 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=45, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5542235393805511]
   [junit4]   2> 78480 T445 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@33a841 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1f5f137),segFN=segments_1,generation=1}
   [junit4]   2> 78480 T445 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 78484 T445 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 78484 T445 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 78484 T445 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 78484 T445 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 78485 T445 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 78485 T445 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 78485 T445 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 78486 T445 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 78486 T445 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 78486 T445 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 78487 T445 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 78487 T445 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 78487 T445 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 78487 T445 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 78487 T445 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 78487 T445 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 78488 T445 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 78494 T445 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 78498 T445 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 78498 T445 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 78499 T445 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=16, maxMergedSegmentMB=5.0927734375, floorSegmentMB=0.62890625, forceMergeDeletesPctAllowed=14.302540920360697, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 78499 T445 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@33a841 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1f5f137),segFN=segments_1,generation=1}
   [junit4]   2> 78499 T445 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 78499 T445 oass.SolrIndexSearcher.<init> Opening Searcher@103407a[collection1] main
   [junit4]   2> 78503 T446 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@103407a[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 78507 T445 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 78509 T398 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 78509 T398 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 78515 T449 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:64792/sv collection:collection1 shard:shard1
   [junit4]   2> 78518 T449 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 78548 T449 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 78554 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 78554 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 78554 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 78560 T449 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 78560 T449 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C113 name=collection1 org.apache.solr.core.SolrCore@3c7d0b url=https://127.0.0.1:64792/sv/collection1 node=127.0.0.1:64792_sv C113_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:64792/sv, core=collection1, node_name=127.0.0.1:64792_sv}
   [junit4]   2> 78560 T449 C113 P64792 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:64792/sv/collection1/
   [junit4]   2> 78560 T449 C113 P64792 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 78560 T449 C113 P64792 oasc.SyncStrategy.syncToMe https://127.0.0.1:64792/sv/collection1/ has no replicas
   [junit4]   2> 78560 T449 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:64792/sv/collection1/ shard1
   [junit4]   2> 78560 T449 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 78574 T423 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 78628 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 78646 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 78656 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 78760 T431 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 78761 T422 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 78762 T444 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 78802 T449 oasc.ZkController.register We are https://127.0.0.1:64792/sv/collection1/ and leader is https://127.0.0.1:64792/sv/collection1/
   [junit4]   2> 78802 T449 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:64792/sv
   [junit4]   2> 78802 T449 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 78802 T449 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 78802 T449 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 78807 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 78807 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 78807 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 78810 T449 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 78813 T423 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 78816 T423 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:64792/sv",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:64792_sv",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 78823 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 78939 T444 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 78940 T431 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 78940 T422 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 79075 T398 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 79076 T398 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 79088 T398 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 79092 T398 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:64801
   [junit4]   2> 79096 T398 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 79096 T398 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 79097 T398 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.RecoveryZkTest-jetty2-1392655619874
   [junit4]   2> 79097 T398 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.RecoveryZkTest-jetty2-1392655619874\'
   [junit4]   2> 79125 T398 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.RecoveryZkTest-jetty2-1392655619874\solr.xml
   [junit4]   2> 79148 T398 oasc.CoreContainer.<init> New CoreContainer 22453648
   [junit4]   2> 79148 T398 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.RecoveryZkTest-jetty2-1392655619874\]
   [junit4]   2> 79150 T398 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 79150 T398 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 79150 T398 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 79150 T398 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 79150 T398 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 79150 T398 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 79150 T398 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 79150 T398 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 79150 T398 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 79160 T398 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 79160 T398 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 79161 T398 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 79161 T398 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:64770/solr
   [junit4]   2> 79163 T398 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 79164 T398 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 79170 T460 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f748b name:ZooKeeperConnection Watcher:127.0.0.1:64770 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 79171 T398 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 79175 T400 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14440be423e0008, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 79177 T398 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 79180 T462 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15bfbf0 name:ZooKeeperConnection Watcher:127.0.0.1:64770/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 79181 T398 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 79190 T398 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 80195 T398 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64801_sv
   [junit4]   2> 80197 T398 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:64801_sv
   [junit4]   2> 80204 T444 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 80206 T431 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 80206 T422 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 80208 T462 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 80218 T463 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 80218 T463 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 80221 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 80221 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 80221 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 80221 T463 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 80224 T423 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 80225 T423 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:64801/sv",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:64801_sv",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 80226 T423 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
   [junit4]   2> 80226 T423 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 80231 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 80360 T431 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 80360 T462 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 80360 T444 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 80360 T422 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 81222 T463 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 81222 T463 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.RecoveryZkTest-jetty2-1392655619874\collection1
   [junit4]   2> 81222 T463 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 81224 T463 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 81224 T463 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 81226 T463 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 81226 T463 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.RecoveryZkTest-jetty2-1392655619874\collection1\'
   [junit4]   2> 81229 T463 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1392655619874/collection1/lib/classes/' to classloader
   [junit4]   2> 81231 T463 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1392655619874/collection1/lib/README' to classloader
   [junit4]   2> 81275 T463 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 81292 T463 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 81297 T463 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 81303 T463 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 81433 T463 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 81434 T463 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 81435 T463 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 81440 T463 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 81496 T463 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 81497 T463 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.RecoveryZkTest-jetty2-1392655619874\collection1\, dataDir=.\org.apache.solr.cloud.RecoveryZkTest-1392655614515/jetty2\
   [junit4]   2> 81497 T463 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@76db10
   [junit4]   2> 81497 T463 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.RecoveryZkTest-1392655614515/jetty2\
   [junit4]   2> 81497 T463 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=.\org.apache.solr.cloud.RecoveryZkTest-1392655614515/jetty2\index/
   [junit4]   2> 81497 T463 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '.\org.apache.solr.cloud.RecoveryZkTest-1392655614515\jetty2\index' doesn't exist. Creating new index...
   [junit4]   2> 81499 T463 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.RecoveryZkTest-1392655614515/jetty2\index
   [junit4]   2> 81500 T463 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=45, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5542235393805511]
   [junit4]   2> 81500 T463 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1239d05 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@13a8dfb),segFN=segments_1,generation=1}
   [junit4]   2> 81500 T463 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 81504 T463 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 81505 T463 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 81505 T463 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 81505 T463 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 81505 T463 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 81506 T463 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 81506 T463 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 81506 T463 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 81506 T463 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 81507 T463 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 81507 T463 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 81507 T463 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 81508 T463 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 81508 T463 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 81508 T463 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 81509 T463 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 81509 T463 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 81515 T463 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 81519 T463 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 81519 T463 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 81520 T463 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=16, maxMergedSegmentMB=5.0927734375, floorSegmentMB=0.62890625, forceMergeDeletesPctAllowed=14.302540920360697, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 81521 T463 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1239d05 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@13a8dfb),segFN=segments_1,generation=1}
   [junit4]   2> 81521 T463 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 81521 T463 oass.SolrIndexSearcher.<init> Opening Searcher@e14a16[collection1] main
   [junit4]   2> 81526 T464 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@e14a16[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 81529 T463 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 81529 T398 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 81529 T398 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 81529 T467 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:64801/sv collection:collection1 shard:shard1
   [junit4]   2> 81536 T398 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 81539 T467 oasc.ZkController.register We are https://127.0.0.1:64801/sv/collection1/ and leader is https://127.0.0.1:64792/sv/collection1/
   [junit4]   2> 81539 T467 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:64801/sv
   [junit4]   2> 81539 T467 oasc.ZkController.checkRecovery Core needs to recover:collection1
   [junit4]   2> 81541 T467 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
   [junit4]   2> ASYNC  NEW_CORE C114 name=collection1 org.apache.solr.core.SolrCore@18447d2 url=https://127.0.0.1:64801/sv/collection1 node=127.0.0.1:64801_sv C114_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:64801/sv, core=collection1, node_name=127.0.0.1:64801_sv}
   [junit4]   2> 81551 T470 C114 P64801 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
   [junit4]   2> 81552 T470 C114 P64801 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
   [junit4]   2> 81552 T470 C114 P64801 oasc.ZkController.publish publishing core=collection1 state=recovering
   [junit4]   2> 81552 T470 C114 P64801 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 81564 T467 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C115 name=collection1 org.apache.solr.core.SolrCore@c7c049 url=https://127.0.0.1:64780/sv/collection1 node=127.0.0.1:64780_sv C115_STATE=coll:control_collection core:collection1 props:{state=active, base_url=https://127.0.0.1:64780/sv, core=collection1, node_name=127.0.0.1:64780_sv, leader=true}
   [junit4]   2> 81679 T414 C115 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-0 (1460305262576402432)]} 0 23
   [junit4]   2> 81692 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 81692 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 81692 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 81760 T398 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 64801
   [junit4]   2> 81760 T398 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=22453648
   [junit4]   2> 81760 T398 oasc.RecoveryStrategy.close WARN Stopping recovery for zkNodeName=core_node2core=collection1
   [junit4]   2> 81766 T423 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 81827 T423 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:64801/sv",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:64801_sv",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 81865 T422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 81867 T418 C115 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-0 (1460305262776680448)]} 0 4
   [junit4]   2> 81876 T444 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 81877 T431 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 81878 T462 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 81878 T422 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> ASYNC  NEW_CORE C116 name=collection1 org.apache.solr.core.SolrCore@3c7d0b url=https://127.0.0.1:64792/sv/collection1 node=127.0.0.1:64792_sv C116_STATE=coll:collection1 core:collection1 props:{state=active, base_url=https://127.0.0.1:64792/sv, core=collection1, node_name=127.0.0.1:64792_sv, leader=true}
   [junit4]   2> 81889 T435 C116 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-0 (1460305262786117632)]} 0 28
   [junit4]   2> 81892 T437 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true
   [junit4]   2> 81932 T437 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 0 seconds.
   [junit4]   2> 81933 T437 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:64801_sv&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node2&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=41 
   [junit4]   2> 81968 T413 C115 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-1 (1460305262885732352)]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C117 name=collection1 org.apache.solr.core.SolrCore@18447d2 url=https://127.0.0.1:64801/sv/collection1 node=127.0.0.1:64801_sv C117_STATE=coll:collection1 core:collection1 props:{state=recovering, base_url=https://127.0.0.1:64801/sv, core=collection1, node_name=127.0.0.1:64801_sv}
   [junit4]   2> 82137 T458 C117 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[2-1 (1460305262902509568)]} 0 10
   [junit4]   2> 82139 T438 C116 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-1 (1460305262902509568)]} 0 155
   [junit4]   2> 82146 T455 C117 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[1-0 (1460305262863712256)]} 0 4
   [junit4]   2> 82151 T437 C116 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-0 (1460305262863712256)]} 0 212
   [junit4]   2> 82173 T415 C115 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-2 (1460305263099641856)]} 0 2
   [junit4]   2> 82182 T417 C115 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-1 (1460305263109079040)]} 0 2
   [junit4]   2> 82276 T457 C117 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[2-2 (1460305263130050560)]} 0 5
   [junit4]   2> 82279 T435 C116 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-2 (1460305263130050560)]} 0 78
   [junit4]   2> 82297 T453 C117 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[1-1 (1460305263200305152)]} 0 0
   [junit4]   2> 82308 T438 C116 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-1 (1460305263200305152)]} 0 47
   [junit4]   2> 82338 T414 C115 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-3 (1460305263272656896)]} 0 5
   [junit4]   2> 82347 T414 C115 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-2 (1460305263283142656)]} 0 1
   [junit4]   2> 82427 T455 C117 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[2-3 (1460305263310405632)]} 0 1
   [junit4]   2> 82435 T435 C116 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-3 (1460305263310405632)]} 0 63
   [junit4]   2> 82442 T453 C117 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[1-2 (1460305263327182848)]} 0 1
   [junit4]   2> 82445 T439 C116 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-2 (1460305263327182848)]} 0 56
   [junit4]   2> 82468 T417 C115 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-3 (1460305263408971776)]} 0 1
   [junit4]   2> 82518 T416 C115 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2&CONTROL=TRUE} {delete=[2-1 (-1460305263461400576)]} 0 1
   [junit4]   2> 82523 T458 C117 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[1-3 (1460305263424700416)]} 0 7
   [junit4]   2> 82529 T437 C116 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-3 (1460305263424700416)]} 0 49
   [junit4]   2> 82561 T455 C117 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {delete=[2-1 (-1460305263484469248)]} 0 1
   [junit4]   2> 82566 T439 C116 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {delete=[2-1 (-1460305263484469248)]} 0 28
   [junit4]   2> 82602 T415 C115 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-4 (1460305263548432384)]} 0 2
   [junit4]   2> ASYNC  NEW_CORE C118 name=collection1 org.apache.solr.core.SolrCore@c7c049 url=https://127.0.0.1:64780/sv/collection1 node=127.0.0.1:64780_sv C118_STATE=coll:control_collection core:collection1 props:{state=active, base_url=https://127.0.0.1:64780/sv, core=collection1, node_name=127.0.0.1:64780_sv, leader=true}
   [junit4]   2> 82658 T418 C118 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2&CONTROL=TRUE} {delete=[1-1 (-1460305263609249792)]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C119 name=collection1 org.apache.solr.core.SolrCore@18447d2 url=https://127.0.0.1:64801/sv/collection1 node=127.0.0.1:64801_sv C119_STATE=coll:collection1 core:collection1 props:{state=recovering, base_url=https://127.0.0.1:64801/sv, core=collection1, node_name=127.0.0.1:64801_sv}
   [junit4]   2> 82985 T458 C119 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {delete=[1-1 (-1460305263633367040)]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C120 name=collection1 org.apache.solr.core.SolrCore@3c7d0b url=https://127.0.0.1:64792/sv/collection1 node=127.0.0.1:64792_sv C120_STATE=coll:collection1 core:collection1 props:{state=active, base_url=https://127.0.0.1:64792/sv, core=collection1, node_name=127.0.0.1:64792_sv, leader=true}
   [junit4]   2> 82987 T435 C120 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {delete=[1-1 (-1460305263633367040)]} 0 307
   [junit4]   2> 82994 T456 C119 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[2-4 (1460305263651192832)]} 0 0
   [junit4]   2> 82997 T439 C120 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-4 (1460305263651192832)]} 0 299
   [junit4]   2> 83011 T413 C118 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-4 (1460305263979397120)]} 0 2
   [junit4]   2> 83057 T457 C119 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[1-4 (1460305263994077184)]} 0 2
   [junit4]   2> 83064 T436 C120 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-4 (1460305263994077184)]} 0 39
   [junit4]   2> 83070 T414 C118 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2&CONTROL=TRUE} {delete=[2-2 (-1460305264041263104)]} 0 3
   [junit4]   2> 83157 T458 C119 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {delete=[2-2 (-1460305264059088896)]} 0 1
   [junit4]   2> 83159 T435 C120 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {delete=[2-2 (-1460305264059088896)]} 0 72
   [junit4]   2> 83162 T417 C118 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2&CONTROL=TRUE} {delete=[1-2 (-1460305264137732096)]} 0 1
   [junit4]   2> 83183 T418 C118 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-5 (1460305264159752192)]} 0 0
   [junit4]   2> 83247 T456 C119 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {delete=[1-2 (-1460305264171286528)]} 0 1
   [junit4]   2> 83249 T440 C120 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {delete=[1-2 (-1460305264171286528)]} 0 56
   [junit4]   2> 83265 T453 C119 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[2-5 (1460305264225812480)]} 0 0
   [junit4]   2> 83268 T435 C120 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-5 (1460305264225812480)]} 0 22
   [junit4]   2> 83271 T416 C118 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-5 (1460305264245735424)]} 0 6
   [junit4]   2> 83300 T454 C119 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[1-5 (1460305264268804096)]} 0 2
   [junit4]   2> 83303 T438 C120 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-5 (1460305264268804096)]} 0 16
   [junit4]   2> 83310 T414 C118 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-6 (1460305264292921344)]} 0 1
   [junit4]   2> 83327 T415 C118 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-6 (1460305264310747136)]} 0 1
   [junit4]   2> 83494 T455 C119 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[2-6 (1460305264312844288)]} 0 5
   [junit4]   2> 83497 T437 C120 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-6 (1460305264312844288)]} 0 168
   [junit4]   2> 83520 T453 C119 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[1-6 (1460305264485859328)]} 0 1
   [junit4]   2> 83522 T435 C120 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-6 (1460305264485859328)]} 0 28
   [junit4]   2> 83546 T416 C118 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-7 (1460305264540385280)]} 0 1
   [junit4]   2> 83572 T414 C118 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-7 (1460305264567648256)]} 0 1
   [junit4]   2> 83580 T458 C119 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[2-7 (1460305264550871040)]} 0 1
   [junit4]   2> 83582 T439 C120 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-7 (1460305264550871040)]} 0 26
   [junit4]   2> 83606 T457 C119 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[1-7 (1460305264587571200)]} 0 1
   [junit4]   2> 83610 T439 C120 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-7 (1460305264587571200)]} 0 19
   [junit4]   2> 83650 T417 C118 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-8 (1460305264649437184)]} 0 1
   [junit4]   2> 83683 T418 C118 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-8 (1460305264682991616)]} 0 2
   [junit4]   2> 83690 T458 C119 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[2-8 (1460305264664117248)]} 0 1
   [junit4]   2> 83692 T440 C120 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-8 (1460305264664117248)]} 0 28
   [junit4]   2> 83719 T455 C119 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[1-8 (1460305264701865984)]} 0 9
   [junit4]   2> 83720 T439 C120 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-8 (1460305264701865984)]} 0 21
   [junit4]   2> 83804 T414 C118 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-9 (1460305264809869312)]} 0 1
   [junit4]   2> 83845 T458 C119 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[2-9 (1460305264824549376)]} 0 0
   [junit4]   2> 83848 T437 C120 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[2-9 (1460305264824549376)]} 0 32
   [junit4]   2> 83861 T415 C118 P64780 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-9 (1460305264869638144)]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C121 name=collection1 org.apache.solr.core.SolrCore@18447d2 url=https://127.0.0.1:64801/sv/collection1 node=127.0.0.1:64801_sv C121_STATE=coll:collection1 core:collection1 props:{state=recovering, base_url=https://127.0.0.1:64801/sv, core=collection1, node_name=127.0.0.1:64801_sv}
   [junit4]   2> 83897 T458 C121 P64801 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:64792/sv/collection1/&wt=javabin&version=2} {add=[1-9 (1460305264897949696)]} 0 0
   [junit4]   2> ASYNC  NEW_CORE C122 name=collection1 org.apache.solr.core.SolrCore@3c7d0b url=https://127.0.0.1:64792/sv/collection1 node=127.0.0.1:64792_sv C122_STATE=coll:collection1 core:collection1 props:{state=active, base_url=https://127.0.0.1:64792/sv, core=collection1, node_name=127.0.0.1:64792_sv, leader=true}
   [junit4]   2> 83899 T440 C122 P64792 oasup.LogUpdateProcessor.finish [collection1] webapp=/sv path=/update params={wt=javabin&version=2} {add=[1-9 (1460305264897949696)]} 0 12
   [junit4]   2> ASYNC  NEW_CORE C123 name=colle

[...truncated too long message...]

 [junit4]   2> 	        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
   [junit4]   2> 	        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 	  15) Thread[id=398, name=TEST-RecoveryZkTest.testDistribSearch-seed#[41014895D1172F7D], state=RUNNABLE, group=TGRP-RecoveryZkTest]
   [junit4]   2> 	        at java.lang.Throwable.fillInStackTrace(Native Method)
   [junit4]   2> 	        at java.lang.Throwable.fillInStackTrace(Throwable.java:783)
   [junit4]   2> 	        at java.lang.Throwable.<init>(Throwable.java:250)
   [junit4]   2> 	        at java.lang.Exception.<init>(Exception.java:54)
   [junit4]   2> 	        at java.lang.InterruptedException.<init>(InterruptedException.java:57)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2067)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1465)
   [junit4]   2> 	        at org.apache.solr.common.util.ExecutorUtil.shutdownAndAwaitTermination(ExecutorUtil.java:54)
   [junit4]   2> 	        at org.apache.solr.update.UpdateShardHandler.close(UpdateShardHandler.java:75)
   [junit4]   2> 	        at org.apache.solr.core.CoreContainer.shutdown(CoreContainer.java:402)
   [junit4]   2> 	        at org.apache.solr.servlet.SolrDispatchFilter.destroy(SolrDispatchFilter.java:210)
   [junit4]   2> 	        at org.apache.solr.cloud.ChaosMonkey.stopJettySolrRunner(ChaosMonkey.java:224)
   [junit4]   2> 	        at org.apache.solr.cloud.ChaosMonkey.stop(ChaosMonkey.java:563)
   [junit4]   2> 	        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.destroyServers(AbstractFullDistribZkTestBase.java:1586)
   [junit4]   2> 	        at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:871)
   [junit4]   2> 	        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> 	        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	        at java.lang.reflect.Method.invoke(Method.java:483)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
   [junit4]   2> 	        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 	        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> NOTE: test params are: codec=Asserting, sim=DefaultSimilarity, locale=es_PA, timezone=America/Los_Angeles
   [junit4]   2> NOTE: Windows 7 6.1 x86/Oracle Corporation 1.8.0 (32-bit)/cpus=2,threads=21,free=75863024,total=123240448
   [junit4]   2> NOTE: All tests run in this JVM: [LukeRequestHandlerTest, TestSolrDeletionPolicy2, TestCSVResponseWriter, TestJmxMonitoredMap, DistributedSpellCheckComponentTest, DistributedQueryComponentCustomSortTest, SuggesterFSTTest, ResourceLoaderTest, DeleteShardTest, FieldAnalysisRequestHandlerTest, SimplePostToolTest, ConnectionManagerTest, FileBasedSpellCheckerTest, UpdateParamsTest, StandardRequestHandlerTest, SpellCheckComponentTest, TestSolrCoreProperties, HighlighterTest, RecoveryZkTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=RecoveryZkTest -Dtests.seed=41014895D1172F7D -Dtests.slow=true -Dtests.locale=es_PA -Dtests.timezone=America/Los_Angeles -Dtests.file.encoding=Cp1252
   [junit4] ERROR   0.00s | RecoveryZkTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([41014895D1172F7D]:0)
   [junit4] Completed in 7268.83s, 1 test, 2 errors <<< FAILURES!

[...truncated 1066 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:453: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:433: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:39: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-targets.xml:37: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build.xml:189: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\common-build.xml:491: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:1268: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:901: There were test failures: 372 suites, 1604 tests, 1 suite-level error, 1 error, 1433 ignored

Total time: 154 minutes 45 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.8.0-fcs-b129 -server -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Windows (32bit/jdk1.7.0_60-ea-b04) - Build # 3789 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/3789/
Java: 32bit/jdk1.7.0_60-ea-b04 -client -XX:+UseG1GC

3 tests failed.
REGRESSION:  org.apache.solr.cloud.BasicDistributedZkTest.testDistribSearch

Error Message:
Timeout occured while waiting response from server at: https://127.0.0.1:65031/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:65031/collection1
	at __randomizedtesting.SeedInfo.seed([395729528B963160:B8B1A74AFCC9515C]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:503)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
	at org.apache.solr.client.solrj.SolrServer.commit(SolrServer.java:168)
	at org.apache.solr.client.solrj.SolrServer.commit(SolrServer.java:146)
	at org.apache.solr.cloud.BasicDistributedZkTest.testUpdateProcessorsRunOnlyOnce(BasicDistributedZkTest.java:663)
	at org.apache.solr.cloud.BasicDistributedZkTest.doTest(BasicDistributedZkTest.java:372)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:152)
	at java.net.SocketInputStream.read(SocketInputStream.java:122)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
	at sun.security.ssl.InputRecord.read(InputRecord.java:480)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:395)
	... 47 more


FAILED:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.testDistribSearch

Error Message:
document count mismatch.  control=979 sum(shards)=980 cloudClient=980

Stack Trace:
java.lang.AssertionError: document count mismatch.  control=979 sum(shards)=980 cloudClient=980
	at __randomizedtesting.SeedInfo.seed([395729528B963160:B8B1A74AFCC9515C]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1230)
	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:213)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testDistribSearch

Error Message:
Timeout occured while waiting response from server at: https://127.0.0.1:52074/bh/g/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:52074/bh/g/collection1
	at __randomizedtesting.SeedInfo.seed([395729528B963160:B8B1A74AFCC9515C]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:503)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.index_specific(AbstractFullDistribZkTestBase.java:698)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingWithSuss(FullSolrCloudDistribCmdsTest.java:443)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.doTest(FullSolrCloudDistribCmdsTest.java:134)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:152)
	at java.net.SocketInputStream.read(SocketInputStream.java:122)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
	at sun.security.ssl.InputRecord.read(InputRecord.java:480)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:395)
	... 48 more




Build Log:
[...truncated 10033 lines...]
   [junit4] Suite: org.apache.solr.cloud.BasicDistributedZkTest
   [junit4]   2> 462585 T1589 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 462585 T1589 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 462589 T1589 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\solrtest-BasicDistributedZkTest-1392632623918
   [junit4]   2> 462591 T1589 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 462595 T1590 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 462691 T1589 oasc.ZkTestServer.run start zk server on port:65009
   [junit4]   2> 462695 T1589 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 462707 T1596 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1aec393 name:ZooKeeperConnection Watcher:127.0.0.1:65009 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 462708 T1589 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 462708 T1589 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 462721 T1591 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1443f5f733a0000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 462724 T1589 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 462735 T1598 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@177da31 name:ZooKeeperConnection Watcher:127.0.0.1:65009/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 462736 T1589 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 462736 T1589 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 462749 T1589 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 462759 T1589 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 462767 T1589 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 462777 T1589 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 462777 T1589 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 462793 T1589 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 462793 T1589 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 462805 T1589 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 462805 T1589 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 462816 T1589 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 462816 T1589 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 462828 T1589 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 462828 T1589 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 462838 T1589 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 462840 T1589 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 462850 T1589 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 462850 T1589 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 462860 T1589 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 462862 T1589 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 462872 T1589 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 462872 T1589 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 462881 T1589 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 462881 T1589 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 462892 T1589 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 462892 T1589 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 462910 T1589 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 462916 T1600 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@63f862 name:ZooKeeperConnection Watcher:127.0.0.1:65009/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 462917 T1589 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 463479 T1589 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 463489 T1589 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 463494 T1589 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:65019
   [junit4]   2> 463497 T1589 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 463497 T1589 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 463497 T1589 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1392632624252
   [junit4]   2> 463498 T1589 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1392632624252\'
   [junit4]   2> 463529 T1589 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1392632624252\solr.xml
   [junit4]   2> 463617 T1589 oasc.CoreContainer.<init> New CoreContainer 11860846
   [junit4]   2> 463618 T1589 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1392632624252\]
   [junit4]   2> 463618 T1589 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 463619 T1589 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 463619 T1589 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 463620 T1589 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 463620 T1589 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 463620 T1589 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 463621 T1589 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 463621 T1589 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 463621 T1589 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 463633 T1589 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 463633 T1589 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 463633 T1589 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 463633 T1589 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:65009/solr
   [junit4]   2> 463634 T1589 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 463636 T1589 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 463645 T1611 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@263e7e name:ZooKeeperConnection Watcher:127.0.0.1:65009 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 463646 T1589 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 463649 T1591 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1443f5f733a0003, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 463652 T1589 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 463655 T1613 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1351311 name:ZooKeeperConnection Watcher:127.0.0.1:65009/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 463656 T1589 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 463661 T1589 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 463671 T1589 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 463679 T1589 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 463683 T1589 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65019_
   [junit4]   2> 463684 T1589 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:65019_
   [junit4]   2> 463691 T1589 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 463695 T1589 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 463703 T1589 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:65019_
   [junit4]   2> 463704 T1589 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 463708 T1589 oasc.Overseer.start Overseer (id=91267571641876484-127.0.0.1:65019_-n_0000000000) starting
   [junit4]   2> 463713 T1589 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 463720 T1615 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 463721 T1589 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 463723 T1615 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 463725 T1589 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 463729 T1589 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 463735 T1614 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 463740 T1616 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 463740 T1616 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 463742 T1616 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 463742 T1613 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 463745 T1614 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 463746 T1614 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:65019",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:65019_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 463746 T1614 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 463746 T1614 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 463752 T1613 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 463754 T1613 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 464623 T1616 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 464623 T1616 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1392632624252\collection1
   [junit4]   2> 464623 T1616 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 464627 T1616 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 464627 T1616 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 464630 T1616 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 464630 T1616 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1392632624252\collection1\'
   [junit4]   2> 464631 T1616 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1392632624252/collection1/lib/classes/' to classloader
   [junit4]   2> 464634 T1616 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1392632624252/collection1/lib/README' to classloader
   [junit4]   2> 464687 T1616 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 464730 T1616 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 464732 T1616 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 464741 T1616 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 465156 T1616 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 465164 T1616 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 465168 T1616 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 465189 T1616 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 465194 T1616 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 465197 T1616 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 465200 T1616 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 465201 T1616 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 465201 T1616 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 465204 T1616 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 465204 T1616 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 465204 T1616 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 465206 T1616 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1392632624252\collection1\, dataDir=.\org.apache.solr.cloud.BasicDistributedZkTest-1392632623917/control/data\
   [junit4]   2> 465206 T1616 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@80278f
   [junit4]   2> 465207 T1616 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.BasicDistributedZkTest-1392632623917/control/data\
   [junit4]   2> 465207 T1616 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=.\org.apache.solr.cloud.BasicDistributedZkTest-1392632623917/control/data\index/
   [junit4]   2> 465207 T1616 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '.\org.apache.solr.cloud.BasicDistributedZkTest-1392632623917\control\data\index' doesn't exist. Creating new index...
   [junit4]   2> 465208 T1616 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.BasicDistributedZkTest-1392632623917/control/data\index
   [junit4]   2> 465208 T1616 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@1825cc5
   [junit4]   2> 465209 T1616 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@f9f7c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@198a3ee),segFN=segments_1,generation=1}
   [junit4]   2> 465210 T1616 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 465214 T1616 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 465215 T1616 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 465216 T1616 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 465216 T1616 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 465217 T1616 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 465217 T1616 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 465218 T1616 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 465218 T1616 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 465219 T1616 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 465220 T1616 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 465221 T1616 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 465221 T1616 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 465222 T1616 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 465223 T1616 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 465223 T1616 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 465223 T1616 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 465224 T1616 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 465235 T1616 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 465242 T1616 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 465242 T1616 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 465244 T1616 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=18, maxMergeAtOnceExplicit=34, maxMergedSegmentMB=1.6728515625, floorSegmentMB=2.0146484375, forceMergeDeletesPctAllowed=5.446957288262679, segmentsPerTier=19.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 465245 T1616 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@f9f7c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@198a3ee),segFN=segments_1,generation=1}
   [junit4]   2> 465245 T1616 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 465245 T1616 oass.SolrIndexSearcher.<init> Opening Searcher@1f20591[collection1] main
   [junit4]   2> 465250 T1617 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1f20591[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 465253 T1616 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 465254 T1589 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 465255 T1589 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 465255 T1620 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:65019 collection:control_collection shard:shard1
   [junit4]   2> 465257 T1620 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 465264 T1589 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 465267 T1622 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f2a92c name:ZooKeeperConnection Watcher:127.0.0.1:65009/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 465267 T1589 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 465269 T1589 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 465270 T1620 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 465273 T1613 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 465274 T1589 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 465275 T1620 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 465275 T1620 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C667 name=collection1 org.apache.solr.core.SolrCore@129fb36 url=https://127.0.0.1:65019/collection1 node=127.0.0.1:65019_ C667_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:65019, core=collection1, node_name=127.0.0.1:65019_}
   [junit4]   2> 465275 T1620 C667 P65019 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:65019/collection1/
   [junit4]   2> 465275 T1620 C667 P65019 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 465275 T1620 C667 P65019 oasc.SyncStrategy.syncToMe https://127.0.0.1:65019/collection1/ has no replicas
   [junit4]   2> 465275 T1620 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:65019/collection1/ shard1
   [junit4]   2> 465275 T1620 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 465279 T1614 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 465288 T1613 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 465290 T1622 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 465294 T1613 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 465302 T1614 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 465307 T1613 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 465398 T1613 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 465398 T1622 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 465429 T1620 oasc.ZkController.register We are https://127.0.0.1:65019/collection1/ and leader is https://127.0.0.1:65019/collection1/
   [junit4]   2> 465429 T1620 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:65019
   [junit4]   2> 465429 T1620 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 465429 T1620 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 465429 T1620 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 465431 T1613 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 465431 T1613 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 465432 T1613 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 465434 T1620 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 465435 T1614 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 465436 T1614 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:65019",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:65019_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 465440 T1613 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 465532 T1613 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 465532 T1622 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 465774 T1589 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 465775 T1589 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 465785 T1589 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 465787 T1589 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:65031
   [junit4]   2> 465794 T1589 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 465794 T1589 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 465794 T1589 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1392632626602
   [junit4]   2> 465794 T1589 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1392632626602\'
   [junit4]   2> 465828 T1589 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1392632626602\solr.xml
   [junit4]   2> 465886 T1589 oasc.CoreContainer.<init> New CoreContainer 26694725
   [junit4]   2> 465887 T1589 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1392632626602\]
   [junit4]   2> 465889 T1589 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 465889 T1589 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 465889 T1589 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 465890 T1589 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 465890 T1589 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 465890 T1589 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 465890 T1589 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 465890 T1589 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 465890 T1589 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 465900 T1589 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 465900 T1589 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 465900 T1589 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 465901 T1589 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:65009/solr
   [junit4]   2> 465901 T1589 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 465903 T1589 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 465910 T1633 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@156759f name:ZooKeeperConnection Watcher:127.0.0.1:65009 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 465911 T1589 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 465915 T1591 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1443f5f733a0006, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 465917 T1589 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 465920 T1635 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5993e3 name:ZooKeeperConnection Watcher:127.0.0.1:65009/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 465920 T1589 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 465930 T1589 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 466804 T1589 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65031_
   [junit4]   2> 466807 T1589 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:65031_
   [junit4]   2> 466819 T1635 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 466819 T1613 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 466819 T1622 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 466841 T1636 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 466843 T1636 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 466846 T1613 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 466846 T1613 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 466847 T1613 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 466846 T1636 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 466854 T1614 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 466855 T1614 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:65031",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:65031_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 466855 T1614 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 466855 T1614 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 466866 T1613 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 466959 T1622 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 466960 T1613 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 466959 T1635 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 467712 T1636 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 467713 T1636 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1392632626602\collection1
   [junit4]   2> 467713 T1636 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 467716 T1636 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 467716 T1636 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 467719 T1636 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 467719 T1636 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1392632626602\collection1\'
   [junit4]   2> 467722 T1636 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1392632626602/collection1/lib/classes/' to classloader
   [junit4]   2> 467722 T1636 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1392632626602/collection1/lib/README' to classloader
   [junit4]   2> 467789 T1636 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 467839 T1636 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 467842 T1636 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 467852 T1636 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 468274 T1636 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 468289 T1636 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 468292 T1636 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 468325 T1636 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 468335 T1636 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 468345 T1636 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 468348 T1636 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 468350 T1636 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 468350 T1636 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 468353 T1636 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 468353 T1636 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 468354 T1636 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 468355 T1636 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1392632626602\collection1\, dataDir=.\org.apache.solr.cloud.BasicDistributedZkTest-1392632623917/jetty1\
   [junit4]   2> 468355 T1636 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@80278f
   [junit4]   2> 468357 T1636 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.BasicDistributedZkTest-1392632623917/jetty1\
   [junit4]   2> 468357 T1636 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=.\org.apache.solr.cloud.BasicDistributedZkTest-1392632623917/jetty1\index/
   [junit4]   2> 468357 T1636 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '.\org.apache.solr.cloud.BasicDistributedZkTest-1392632623917\jetty1\index' doesn't exist. Creating new index...
   [junit4]   2> 468358 T1636 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.BasicDistributedZkTest-1392632623917/jetty1\index
   [junit4]   2> 468358 T1636 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@9afa78
   [junit4]   2> 468360 T1636 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1ed8437 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@b20dd4),segFN=segments_1,generation=1}
   [junit4]   2> 468360 T1636 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 468368 T1636 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 468369 T1636 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 468370 T1636 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 468370 T1636 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 468372 T1636 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 468372 T1636 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 468373 T1636 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 468374 T1636 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 468374 T1636 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 468374 T1636 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 468375 T1636 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 468375 T1636 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 468375 T1636 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 468376 T1636 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 468376 T1636 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 468398 T1636 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 468399 T1636 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 468405 T1636 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 468411 T1636 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 468412 T1636 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 468412 T1636 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=18, maxMergeAtOnceExplicit=34, maxMergedSegmentMB=1.6728515625, floorSegmentMB=2.0146484375, forceMergeDeletesPctAllowed=5.446957288262679, segmentsPerTier=19.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 468413 T1636 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1ed8437 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@b20dd4),segFN=segments_1,generation=1}
   [junit4]   2> 468413 T1636 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 468414 T1636 oass.SolrIndexSearcher.<init> Opening Searcher@4a6234[collection1] main
   [junit4]   2> 468422 T1636 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 468423 T1637 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4a6234[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 468423 T1589 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 468423 T1589 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 468423 T1640 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:65031 collection:collection1 shard:shard1
   [junit4]   2> 468426 T1640 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 468440 T1640 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 468444 T1613 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 468444 T1613 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 468444 T1613 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 468445 T1640 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 468445 T1640 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C668 name=collection1 org.apache.solr.core.SolrCore@132be9f url=https://127.0.0.1:65031/collection1 node=127.0.0.1:65031_ C668_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:65031, core=collection1, node_name=127.0.0.1:65031_}
   [junit4]   2> 468445 T1640 C668 P65031 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:65031/collection1/
   [junit4]   2> 468445 T1640 C668 P65031 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 468445 T1640 C668 P65031 oasc.SyncStrategy.syncToMe https://127.0.0.1:65031/collection1/ has no replicas
   [junit4]   2> 468446 T1640 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:65031/collection1/ shard1
   [junit4]   2> 468446 T1640 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 468449 T1614 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 468456 T1613 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 468460 T1613 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 468467 T1613 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 468469 T1635 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 468470 T1622 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 468470 T1613 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 468505 T1640 oasc.ZkController.register We are https://127.0.0.1:65031/collection1/ and leader is https://127.0.0.1:65031/collection1/
   [junit4]   2> 468505 T1640 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:65031
   [junit4]   2> 468505 T1640 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 468505 T1640 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 468505 T1640 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 468508 T1613 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 468509 T1640 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 468511 T1614 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 468513 T1614 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:65031",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:65031_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 468517 T1613 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 468607 T1635 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 468608 T1613 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 468608 T1622 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 468916 T1589 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 468917 T1589 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 468927 T1589 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 468929 T1589 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:65040
   [junit4]   2> 468933 T1589 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 468933 T1589 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 468933 T1589 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: .\org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1392632629755
   [junit4]   2> 468933 T1589 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '.\org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1392632629755\'
   [junit4]   2> 468965 T1589 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1392632629755\solr.xml
   [junit4]   2> 469017 T1589 oasc.CoreContainer.<init> New CoreContainer 23481809
   [junit4]   2> 469017 T1589 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=.\org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1392632629755\]
   [junit4]   2> 469019 T1589 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 469019 T1589 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 469019 T1589 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 469019 T1589 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 469019 T1589 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 469019 T1589 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 469019 T1589 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 469019 T1589 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 469019 T1589 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 469030 T1589 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 469030 T1589 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 469030 T1589 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 469030 T1589 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:65009/solr
   [junit4]   2> 469030 T1589 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 469034 T1589 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 469051 T1651 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c93fb6 name:ZooKeeperConnection Watcher:127.0.0.1:65009 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 469052 T1589 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 469056 T1591 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1443f5f733a0008, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 469057 T1589 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 469060 T1653 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@534036 name:ZooKeeperConnection Watcher:127.0.0.1:65009/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 469061 T1589 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 469069 T1589 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 469940 T1589 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65040_
   [junit4]   2> 469945 T1589 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:65040_
   [junit4]   2> 469958 T1635 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 469960 T1653 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 469961 T1622 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 469965 T1613 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 469981 T1654 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 469981 T1654 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 469984 T1654 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 469984 T1613 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 469984 T1613 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 469984 T1613 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 469987 T1614 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 469989 T1614 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:65040",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:65040_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 469989 T1614 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 469989 T1614 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 469994 T1613 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 470087 T1635 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 470087 T1613 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 470087 T1622 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 470087 T1653 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 470848 T1654 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 470848 T1654 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: .\org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1392632629755\collection1
   [junit4]   2> 470848 T1654 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 470852 T1654 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 470852 T1654 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 470855 T1654 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 470855 T1654 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '.\org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1392632629755\collection1\'
   [junit4]   2> 470857 T1654 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1392632629755/collection1/lib/classes/' to classloader
   [junit4]   2> 470857 T1654 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1392632629755/collection1/lib/README' to classloader
   [junit4]   2> 470929 T1654 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 470976 T1654 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 470978 T1654 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 470989 T1654 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 471406 T1654 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 471416 T1654 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 471419 T1654 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 471439 T1654 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 471443 T1654 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 471448 T1654 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 471449 T1654 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 471450 T1654 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 471450 T1654 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 471452 T1654 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 471452 T1654 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 471452 T1654 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 471453 T1654 oasc.SolrCore.<init> [collection1] Opening new SolrCore at .\org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1392632629755\collection1\, dataDir=.\org.apache.solr.cloud.BasicDistributedZkTest-1392632623917/jetty2\
   [junit4]   2> 471453 T1654 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@80278f
   [junit4]   2> 471454 T1654 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.BasicDistributedZkTest-1392632623917/jetty2\
   [junit4]   2> 471454 T1654 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=.\org.apache.solr.cloud.BasicDistributedZkTest-1392632623917/jetty2\index/
   [junit4]   2> 471454 T1654 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '.\org.apache.solr.cloud.BasicDistributedZkTest-1392632623917\jetty2\index' doesn't exist. Creating new index...
   [junit4]   2> 471454 T1654 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.BasicDistributedZkTest-1392632623917/jetty2\index
   [junit4]   2> 471455 T1654 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@94fec1
   [junit4]   2> 471455 T1654 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@627be9 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1e16bac),segFN=segments_1,generation=1}
   [junit4]   2> 471455 T1654 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 471459 T1654 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 471459 T1654 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 471459 T1654 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 471459 T1654 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 471460 T1654 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 471460 T1654 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 471460 T1654 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 471461 T1654 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 471461 T1654 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 471462 T1654 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 471463 T1654 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 471463 T1654 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 471463 T1654 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 471464 T1654 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 471464 T1654 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 471465 T1654 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 471465 T1654 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 471471 T1654 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 471476 T1654 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 471476 T1654 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 471478 T1654 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.

[...truncated too long message...]

n.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 2174254 T6743 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 2174254 T6743 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 2174254 T6743 oasc.CachingDirectoryFactory.closeCacheValue looking to close .\org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1392634236886/jetty5\index [CachedDir<<refCount=0;path=.\org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1392634236886/jetty5\index;done=false>>]
   [junit4]   2> 2174254 T6743 oasc.CachingDirectoryFactory.close Closing directory: .\org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1392634236886/jetty5\index
   [junit4]   2> 2174254 T6743 oasc.CachingDirectoryFactory.closeCacheValue looking to close .\org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1392634236886/jetty5\ [CachedDir<<refCount=0;path=.\org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1392634236886/jetty5\;done=false>>]
   [junit4]   2> 2174254 T6743 oasc.CachingDirectoryFactory.close Closing directory: .\org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1392634236886/jetty5\
   [junit4]   2> 2174254 T6743 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 2174256 T6743 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 2174256 T6743 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 2174256 T6743 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> NOTE: test params are: codec=HighCompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=HIGH_COMPRESSION, chunkSize=468), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=HIGH_COMPRESSION, chunkSize=468)), sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=it_IT, timezone=Pacific/Wake
   [junit4]   2> NOTE: Windows 7 6.1 x86/Oracle Corporation 1.7.0_60-ea (32-bit)/cpus=2,threads=1,free=122631688,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [ChangedSchemaMergeTest, WordBreakSolrSpellCheckerTest, DateMathParserTest, TestHighFrequencyDictionaryFactory, SolrCmdDistributorTest, SchemaVersionSpecificBehaviorTest, TestFiltering, ScriptEngineTest, TestDistributedSearch, TestFileDictionaryLookup, OverseerCollectionProcessorTest, HighlighterConfigTest, TestCustomSort, HdfsCollectionsAPIDistributedZkTest, AnalysisAfterCoreReloadTest, TestCSVLoader, TestCloudManagedSchemaAddField, TestNumberUtils, FieldFacetExtrasTest, EnumFieldTest, TestFuzzyAnalyzedSuggestions, CopyFieldTest, BadIndexSchemaTest, TestPostingsSolrHighlighter, TestCodecSupport, CoreContainerCoreInitFailuresTest, QueryElevationComponentTest, LukeRequestHandlerTest, PreAnalyzedFieldTest, OutputWriterTest, TestFoldingMultitermQuery, TestSerializedLuceneMatchVersion, BlockCacheTest, TestStressReorder, TestReversedWildcardFilterFactory, TestOverriddenPrefixQueryForCustomFieldType, ChaosMonkeySafeLeaderTest, MBeansHandlerTest, ExternalFileFieldSortTest, SuggesterTSTTest, TestCollationFieldDocValues, UnloadDistributedZkTest, SolrIndexSplitterTest, OverseerRolesTest, ShowFileRequestHandlerTest, IndexBasedSpellCheckerTest, TestSearchPerf, UUIDFieldTest, SampleTest, TestBlendedInfixSuggestions, RequestHandlersTest, TestAnalyzeInfixSuggestions, TestRecovery, QueryFacetTest, MultiTermTest, SpellingQueryConverterTest, PreAnalyzedUpdateProcessorTest, StressHdfsTest, BasicDistributedZkTest, TestBinaryField, TestSolrQueryParserResource, TestDynamicFieldCollectionResource, TestSolrXmlPersistor, TestSolr4Spatial, TestNonNRTOpen, TestDocSet, OpenCloseCoreStressTest, TestJmxIntegration, TestReplicationHandler, DOMUtilTest, UniqFieldsUpdateProcessorFactoryTest, HardAutoCommitTest, TestCSVResponseWriter, TestFunctionQuery, TestPartialUpdateDeduplication, UpdateParamsTest, DefaultValueUpdateProcessorTest, SuggesterTest, TestTrie, SolrCoreCheckLockOnStartupTest, HdfsLockFactoryTest, ShardRoutingCustomTest, CacheHeaderTest, CursorMarkTest, TestSort, SearchHandlerTest, DocValuesMissingTest, QueryResultKeyTest, StandardRequestHandlerTest, MigrateRouteKeyTest, TestPhraseSuggestions, SpellPossibilityIteratorTest, DistributedSuggestComponentTest, TestStandardQParsers, TestIndexSearcher, TestRandomMergePolicy, TestUpdate, TestBinaryResponseWriter, TestUtils, AddSchemaFieldsUpdateProcessorFactoryTest, TestModifyConfFiles, TestUniqueKeyFieldResource, SOLR749Test, TestDefaultSimilarityFactory, TestSuggestSpellingConverter, JSONWriterTest, TestRecoveryHdfs, HdfsBasicDistributedZk2Test, TestCoreContainer, RemoteQueryErrorTest, TestComponentsName, PeerSyncTest, DirectSolrConnectionTest, TestOmitPositions, FieldMutatingUpdateProcessorTest, XmlUpdateRequestHandlerTest, JsonLoaderTest, SimpleFacetsTest, LoggingHandlerTest, TestQueryUtils, TestXIncludeConfig, QueryEqualityTest, SpatialFilterTest, TestSchemaVersionResource, SoftAutoCommitTest, TestDynamicFieldResource, TestIndexingPerformance, TestFaceting, TestIBSimilarityFactory, TestSchemaNameResource, ChaosMonkeyNothingIsSafeTest, DistributedQueryElevationComponentTest, TestSolrXmlPersistence, TestRealTimeGet, CoreAdminCreateDiscoverTest, TestRandomDVFaceting, TestFastWriter, HdfsUnloadDistributedZkTest, TestStressRecovery, LegacyHTMLStripCharFilterTest, NotRequiredUniqueKeyTest, TestSolrQueryParser, ModifyConfFileTest, TestBM25SimilarityFactory, TestAddFieldRealTimeGet, BadCopyFieldTest, FileUtilsTest, QueryParsingTest, TestWordDelimiterFilterFactory, SuggesterWFSTTest, ExpressionTest, DocValuesTest, NumericFieldsTest, RequiredFieldsTest, PathHierarchyTokenizerFactoryTest, TestHighlightDedupGrouping, TestReload, BasicZkTest, ResponseLogComponentTest, TestStressUserVersions, TestPseudoReturnFields, TestGroupingSearch, DeleteShardTest, LeaderElectionTest, DocumentBuilderTest, AlternateDirectoryTest, DebugComponentTest, TestPHPSerializedResponseWriter, TestPerFieldSimilarity, TestCharFilters, PingRequestHandlerTest, AbstractAnalyticsStatsTest, DirectSolrSpellCheckerTest, SolrCoreTest, TestRTGBase, TestNoOpRegenerator, SuggestComponentTest, BasicDistributedZk2Test, CollectionsAPIDistributedZkTest, SyncSliceTest, OverseerTest, LeaderElectionIntegrationTest, ShardRoutingTest, RecoveryZkTest, FullSolrCloudDistribCmdsTest]
   [junit4] Completed in 99.76s, 1 test, 1 error <<< FAILURES!

[...truncated 552 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:453: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:433: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:39: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-targets.xml:37: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build.xml:189: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\common-build.xml:491: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:1268: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:901: There were test failures: 372 suites, 1604 tests, 2 errors, 1 failure, 40 ignored (12 assumptions)

Total time: 82 minutes 58 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_60-ea-b04 -client -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_51) - Build # 3788 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/3788/
Java: 64bit/jdk1.7.0_51 -XX:-UseCompressedOops -XX:+UseParallelGC

3 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.testDistribSearch

Error Message:
document count mismatch.  control=890 sum(shards)=889 cloudClient=889

Stack Trace:
java.lang.AssertionError: document count mismatch.  control=890 sum(shards)=889 cloudClient=889
	at __randomizedtesting.SeedInfo.seed([602657E0B516B984:E1C0D9F8C249D9B8]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1230)
	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:213)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)


REGRESSION:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testDistribSearch

Error Message:
Timeout occured while waiting response from server at: https://127.0.0.1:59670/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:59670/collection1
	at __randomizedtesting.SeedInfo.seed([602657E0B516B984:E1C0D9F8C249D9B8]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:503)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.index_specific(AbstractFullDistribZkTestBase.java:698)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingWithSuss(FullSolrCloudDistribCmdsTest.java:443)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.doTest(FullSolrCloudDistribCmdsTest.java:134)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:152)
	at java.net.SocketInputStream.read(SocketInputStream.java:122)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
	at sun.security.ssl.InputRecord.read(InputRecord.java:480)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:395)
	... 47 more


REGRESSION:  org.apache.solr.cloud.OverseerTest.testOverseerFailure

Error Message:
KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1

Stack Trace:
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
	at __randomizedtesting.SeedInfo.seed([602657E0B516B984:642ED813A7B356A5]:0)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:429)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:426)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:383)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:370)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:156)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:153)
	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:584)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 10120 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerTest
   [junit4]   2> 777773 T1683 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 777773 T1683 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> Creating dataDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\solrtest-OverseerTest-1392609499977
   [junit4]   2> 777775 T1683 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 777782 T1683 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignmentBigger
   [junit4]   2> 777782 T1683 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 777783 T1684 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 777863 T1683 oasc.ZkTestServer.run start zk server on port:56234
   [junit4]   2> 777866 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 777871 T1690 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6c44c5ca name:ZooKeeperConnection Watcher:127.0.0.1:56234 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 777872 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 777877 T1685 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1443dfe9b580000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 777878 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 777881 T1692 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@63de0beb name:ZooKeeperConnection Watcher:127.0.0.1:56234 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 777881 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 777881 T1683 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 777889 T1685 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1443dfe9b580001, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 777890 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 777892 T1694 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3fd4becb name:ZooKeeperConnection Watcher:127.0.0.1:56234/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 777893 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 777893 T1683 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 777899 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 777902 T1696 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@58ae2329 name:ZooKeeperConnection Watcher:127.0.0.1:56234/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 777903 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 777904 T1683 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 777909 T1683 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 777917 T1683 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:56234_solr
   [junit4]   2> 777918 T1683 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 777922 T1683 oasc.Overseer.start Overseer (id=91266056191475715-127.0.0.1:56234_solr-n_0000000000) starting
   [junit4]   2> 777925 T1683 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 777930 T1683 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 777936 T1683 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 777940 T1698 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 777941 T1683 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 777942 T1698 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 777943 T1697 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 777946 T1683 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 777949 T1683 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 777955 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 777959 T1700 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@63387629 name:ZooKeeperConnection Watcher:127.0.0.1:56234/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 777960 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 777962 T1683 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 777966 T1683 oascc.SolrZkClient.makePath makePath: /live_nodes/node0
   [junit4]   2> 777972 T1694 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 777972 T1700 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 777973 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 777977 T1702 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@346f357f name:ZooKeeperConnection Watcher:127.0.0.1:56234/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 777977 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 777979 T1683 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 777984 T1683 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 777988 T1702 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 777988 T1700 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 777989 T1694 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 777991 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 777995 T1704 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2f496827 name:ZooKeeperConnection Watcher:127.0.0.1:56234/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 777995 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 777997 T1683 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 778003 T1683 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
   [junit4]   2> 778007 T1702 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 778008 T1700 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 778009 T1704 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 778010 T1694 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 778010 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 778013 T1706 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@28eddfc4 name:ZooKeeperConnection Watcher:127.0.0.1:56234/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 778013 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 778015 T1683 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 778018 T1683 oascc.SolrZkClient.makePath makePath: /live_nodes/node3
   [junit4]   2> 778022 T1706 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 778023 T1704 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 778023 T1694 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 778024 T1702 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 778024 T1700 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 778025 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 778028 T1708 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3008e741 name:ZooKeeperConnection Watcher:127.0.0.1:56234/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 778028 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 778030 T1683 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 778033 T1683 oascc.SolrZkClient.makePath makePath: /live_nodes/node4
   [junit4]   2> 778038 T1708 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 778038 T1700 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 778038 T1702 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 778039 T1706 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 778039 T1694 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 778040 T1704 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 778040 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 778043 T1710 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@79a8567b name:ZooKeeperConnection Watcher:127.0.0.1:56234/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 778043 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 778046 T1683 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 778050 T1683 oascc.SolrZkClient.makePath makePath: /live_nodes/node5
   [junit4]   2> 778057 T1694 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 778058 T1706 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 778058 T1704 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 778058 T1710 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 778059 T1708 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 778060 T1700 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 778060 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 778060 T1702 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 778063 T1712 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1beaced2 name:ZooKeeperConnection Watcher:127.0.0.1:56234/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 778064 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 778066 T1683 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 778069 T1683 oascc.SolrZkClient.makePath makePath: /live_nodes/node6
   [junit4]   2> 778075 T1712 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 778076 T1700 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 778077 T1706 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 778077 T1710 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 778077 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 778078 T1694 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 778078 T1708 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 778078 T1702 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 778078 T1704 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 778081 T1714 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@74709ec3 name:ZooKeeperConnection Watcher:127.0.0.1:56234/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 778082 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 778084 T1683 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 778088 T1683 oascc.SolrZkClient.makePath makePath: /live_nodes/node7
   [junit4]   2> 778093 T1710 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 778095 T1702 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 778095 T1704 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 778095 T1706 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 778095 T1694 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 778095 T1700 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 778095 T1708 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 778095 T1712 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 778095 T1714 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 778096 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 778099 T1716 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c371c72 name:ZooKeeperConnection Watcher:127.0.0.1:56234/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 778100 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 778102 T1683 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 778106 T1683 oascc.SolrZkClient.makePath makePath: /live_nodes/node8
   [junit4]   2> 778111 T1702 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 778112 T1712 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 778113 T1714 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 778113 T1706 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 778114 T1704 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 778114 T1708 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 778114 T1694 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 778114 T1700 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 778115 T1716 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 778116 T1710 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 778117 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 778120 T1718 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@288b99b name:ZooKeeperConnection Watcher:127.0.0.1:56234/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 778120 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 778122 T1683 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 778127 T1683 oascc.SolrZkClient.makePath makePath: /live_nodes/node9
   [junit4]   2> 778132 T1716 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 778135 T1704 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 778135 T1714 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 778135 T1700 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 778135 T1694 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 778136 T1712 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 778136 T1706 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 778136 T1702 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 778136 T1710 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 778136 T1718 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 778136 T1708 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 778137 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 778140 T1720 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@47426835 name:ZooKeeperConnection Watcher:127.0.0.1:56234/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 778141 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 778143 T1683 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 778148 T1683 oascc.SolrZkClient.makePath makePath: /live_nodes/node10
   [junit4]   2> 778154 T1704 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 778154 T1716 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 778156 T1718 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 778157 T1714 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 778158 T1720 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 778158 T1712 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 778158 T1708 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 778158 T1706 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 778158 T1710 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 778158 T1694 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 778158 T1700 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 778159 T1702 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 778160 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 778163 T1722 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2c96cf7b name:ZooKeeperConnection Watcher:127.0.0.1:56234/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 778164 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 778165 T1683 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 778170 T1683 oascc.SolrZkClient.makePath makePath: /live_nodes/node11
   [junit4]   2> 778176 T1716 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 778176 T1714 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 778176 T1708 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 778179 T1694 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 778179 T1702 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 778179 T1722 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 778180 T1712 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 778180 T1700 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 778180 T1706 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 778180 T1704 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 778180 T1718 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 778180 T1710 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 778180 T1720 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 778181 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 778184 T1724 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12241210 name:ZooKeeperConnection Watcher:127.0.0.1:56234/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 778185 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 778187 T1683 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 778191 T1683 oascc.SolrZkClient.makePath makePath: /live_nodes/node12
   [junit4]   2> 778198 T1712 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 778200 T1700 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 778201 T1704 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 778202 T1724 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 778202 T1708 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 778202 T1716 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 778202 T1714 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 778202 T1694 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 778202 T1702 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 778202 T1722 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 778202 T1720 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 778202 T1710 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 778202 T1718 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 778203 T1706 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 778204 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 778207 T1726 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@24df034c name:ZooKeeperConnection Watcher:127.0.0.1:56234/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 778207 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 778209 T1683 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 778214 T1683 oascc.SolrZkClient.makePath makePath: /live_nodes/node13
   [junit4]   2> 778220 T1726 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 778220 T1706 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 778220 T1714 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 778221 T1722 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 778222 T1704 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 778224 T1724 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 778224 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 778224 T1708 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 778224 T1694 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 778224 T1716 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 778225 T1700 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 778225 T1712 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 778225 T1710 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 778225 T1718 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 778226 T1702 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 778226 T1720 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 778229 T1728 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1cb3e9d5 name:ZooKeeperConnection Watcher:127.0.0.1:56234/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 778230 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 778233 T1683 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 778237 T1683 oascc.SolrZkClient.makePath makePath: /live_nodes/node14
   [junit4]   2> 778244 T1714 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 778245 T1724 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 778246 T1700 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 778246 T1728 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 778246 T1704 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 778246 T1706 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 778246 T1718 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 778246 T1710 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 778247 T1726 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 778247 T1722 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 778247 T1720 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 778248 T1716 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 778248 T1702 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 778248 T1708 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 778249 T1694 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 778249 T1712 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 778249 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 778252 T1730 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16c79d8a name:ZooKeeperConnection Watcher:127.0.0.1:56234/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 778252 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 778255 T1683 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 778261 T1683 oascc.SolrZkClient.makePath makePath: /live_nodes/node15
   [junit4]   2> 778269 T1728 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 778271 T1700 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 778271 T1710 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 778271 T1712 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 778271 T1730 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 778272 T1694 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 778272 T1716 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 778272 T1714 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 778272 T1724 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 778272 T1708 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 778272 T1720 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 778272 T1722 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 778272 T1726 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 778272 T1702 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 778273 T1704 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 778273 T1718 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 778274 T1706 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 778274 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 778277 T1732 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@197dd7a0 name:ZooKeeperConnection Watcher:127.0.0.1:56234/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 778277 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 778280 T1683 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 778284 T1683 oascc.SolrZkClient.makePath makePath: /live_nodes/node16
   [junit4]   2> 778291 T1704 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 778291 T1730 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 778291 T1718 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 778291 T1710 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 778292 T1706 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 778292 T1728 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 778292 T1712 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 778292 T1700 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 778294 T1702 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 778294 T1732 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 778294 T1720 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 778294 T1708 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 778294 T1722 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 778294 T1724 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 778294 T1726 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 778295 T1716 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 778295 T1694 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 778295 T1714 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 778296 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 778299 T1734 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4b7dc8f7 name:ZooKeeperConnection Watcher:127.0.0.1:56234/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 778299 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 778302 T1683 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 778305 T1683 oascc.SolrZkClient.makePath makePath: /live_nodes/node17
   [junit4]   2> 778313 T1734 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 778313 T1700 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 778313 T1728 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 778313 T1710 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 778314 T1712 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 778314 T1718 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 778314 T1706 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 778316 T1730 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 778316 T1704 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 778316 T1708 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 778316 T1722 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 778317 T1694 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 778317 T1702 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 778317 T1726 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 778317 T1716 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 778317 T1714 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 778318 T1732 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 778318 T1724 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 778318 T1720 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 778320 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 778324 T1736 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@276153d5 name:ZooKeeperConnection Watcher:127.0.0.1:56234/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 778324 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 778326 T1683 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 778331 T1683 oascc.SolrZkClient.makePath makePath: /live_nodes/node18
   [junit4]   2> 778339 T1728 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 778340 T1712 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 778340 T1730 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 778340 T1716 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 778340 T1718 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 778340 T1704 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 778341 T1706 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 778341 T1710 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 778341 T1732 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 778341 T1702 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 778343 T1700 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 778343 T1720 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 778343 T1694 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 778343 T1734 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 778343 T1722 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 778344 T1708 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 778344 T1726 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 778344 T1724 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 778344 T1714 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 778345 T1736 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 778346 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 778350 T1738 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@28390e54 name:ZooKeeperConnection Watcher:127.0.0.1:56234/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 778350 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 778352 T1683 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 778356 T1683 oascc.SolrZkClient.makePath makePath: /live_nodes/node19
   [junit4]   2> 778364 T1720 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 778364 T1722 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 778364 T1726 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 778364 T1702 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 778364 T1716 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 778364 T1694 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 778364 T1732 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 778365 T1724 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 778365 T1714 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 778365 T1708 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 778367 T1710 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 778368 T1730 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 778368 T1718 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 778368 T1704 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 778368 T1734 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 778369 T1706 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 778369 T1700 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 778370 T1738 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 778370 T1728 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 778370 T1712 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 778371 T1736 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 778371 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 778374 T1740 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5c9799ca name:ZooKeeperConnection Watcher:127.0.0.1:56234/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 778374 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 778376 T1683 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 778380 T1683 oascc.SolrZkClient.makePath makePath: /live_nodes/node20
   [junit4]   2> 778386 T1722 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 778387 T1726 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 778388 T1702 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 778396 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 778397 T1712 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 778398 T1716 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 778398 T1720 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 778398 T1694 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 778398 T1700 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 778398 T1710 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 778398 T1740 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 778398 T1714 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 778398 T1708 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 778399 T1738 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 778399 T1736 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 778399 T1728 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 778400 T1732 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 778400 T1724 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 778400 T1704 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 778400 T1730 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 778401 T1718 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 778401 T1734 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 778401 T1706 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 778406 T1742 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@618d20df name:ZooKeeperConnection Watcher:127.0.0.1:56234/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 778407 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 778409 T1683 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 778413 T1683 oascc.SolrZkClient.makePath makePath: /live_nodes/node21
   [junit4]   2> 778429 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 778429 T1728 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 778431 T1740 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 778431 T1720 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 778432 T1738 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 778432 T1706 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 778433 T1736 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 778433 T1704 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 778433 T1716 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 778433 T1734 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 778433 T1708 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 778433 T1694 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 778434 T1724 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 778434 T1732 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 778434 T1742 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 778434 T1730 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 778434 T1714 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 778434 T1722 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 778434 T1726 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 778434 T1702 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 778435 T1718 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 778435 T1710 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 778435 T1700 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 778435 T1712 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 778439 T1744 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@244b49dd name:ZooKeeperConnection Watcher:127.0.0.1:56234/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 778439 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 778442 T1683 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 778446 T1683 oascc.SolrZkClient.makePath makePath: /live_nodes/node22
   [junit4]   2> 778452 T1738 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 778452 T1728 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 778452 T1700 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 778458 T1740 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 778458 T1736 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 778458 T1712 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 778458 T1710 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 778458 T1744 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 778459 T1716 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 778459 T1694 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 778459 T1708 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 778459 T1724 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 778460 T1714 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 778460 T1732 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 778460 T1726 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 778460 T1720 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 778460 T1722 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 778460 T1742 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 778460 T1702 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 778460 T1704 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 778460 T1734 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 778462 T1706 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 778462 T1730 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 778462 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 778463 T1718 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 778466 T1746 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@635ad3f0 name:ZooKeeperConnection Watcher:127.0.0.1:56234/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 778466 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 778468 T1683 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 778472 T1683 oascc.SolrZkClient.makePath makePath: /live_nodes/node23
   [junit4]   2> 778477 T1708 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 778478 T1702 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 778481 T1734 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 778481 T1704 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 778481 T1732 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 778482 T1744 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 778482 T1700 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 778482 T1722 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 778482 T1718 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 778482 T1724 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 778482 T1746 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 778482 T1728 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 778482 T1740 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 778483 T1712 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 778484 T1736 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 778484 T1738 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 778484 T1730 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 778484 T1706 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 778484 T1710 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 778484 T1720 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 778485 T1726 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 778485 T1742 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 778485 T1694 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 778485 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 778485 T1714 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 778485 T1716 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 778488 T1748 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2f7a6630 name:ZooKeeperConnection Watcher:127.0.0.1:56234/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 778489 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 778490 T1683 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 778494 T1683 oascc.SolrZkClient.makePath makePath: /live_nodes/node24
   [junit4]   2> 778501 T1716 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 778501 T1714 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 778501 T1724 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 778503 T1694 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 778503 T1708 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 778504 T1742 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 778504 T1722 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 778504 T1730 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 778534 T1734 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 778535 T1732 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 778535 T1718 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 778536 T1702 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 778536 T1746 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 778536 T1738 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 778536 T1748 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 778536 T1704 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 778536 T1710 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 778537 T1726 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 778537 T1706 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 778537 T1740 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 778537 T1720 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 778537 T1712 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 778537 T1736 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 778538 T1728 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 778538 T1700 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 778538 T1744 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 778538 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 778540 T1750 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2b0526f0 name:ZooKeeperConnection Watcher:127.0.0.1:56234/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 778541 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 778542 T1683 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 778545 T1683 oascc.SolrZkClient.makePath makePath: /live_nodes/node25
   [junit4]   2> 778555 T1694 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 778555 T1706 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 778555 T1726 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 778555 T1730 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 778556 T1720 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 778556 T1710 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 778556 T1742 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 778556 T1716 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 778556 T1714 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 778556 T1724 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 778556 T1732 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 778559 T1702 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 778559 T1740 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 778559 T1736 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 778559 T1708 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 778559 T1712 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 778559 T1746 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 778560 T1722 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 778560 T1750 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 778560 T1738 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 778560 T1748 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 778560 T1728 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 778560 T1744 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 778560 T1700 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 778560 T1704 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 778560 T1734 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 778560 T1718 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 778560 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 778564 T1752 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@31cb5b8b name:ZooKeeperConnection Watcher:127.0.0.1:56234/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 778564 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 778566 T1683 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 778570 T1683 oascc.SolrZkClient.makePath makePath: /live_nodes/node26
   [junit4]   2> 778578 T1702 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 778578 T1704 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 778578 T1746 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 778578 T1734 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 778578 T1718 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 778578 T1748 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 778579 T1744 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 778579 T1700 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 778580 T1722 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 778582 T1732 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 778582 T1716 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 778582 T1694 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 778582 T1730 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 778582 T1752 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 778582 T1742 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 778582 T1728 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 778582 T1720 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 778582 T1708 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 778582 T1714 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 778582 T1740 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 778582 T1750 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 778582 T1706 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 778582 T1736 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 778582 T1724 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 778582 T1726 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 778582 T1712 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 778582 T1738 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 778582 T1710 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 778584 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 778587 T1754 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@38ab9d4a name:ZooKeeperConnection Watcher:127.0.0.1:56234/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 778587 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 778589 T1683 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 778593 T1683 oascc.SolrZkClient.makePath makePath: /live_nodes/node27
   [junit4]   2> 778598 T1732 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 778599 T1724 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 778601 T1712 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 778604 T1730 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 778604 T1708 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 778604 T1736 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 778604 T1738 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 778604 T1740 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 778604 T1716 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 778604 T1702 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 778604 T1720 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 778604 T1714 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 778604 T1694 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 778604 T1750 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 778605 T1742 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 778605 T1752 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 778605 T1706 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 778605 T1726 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 778605 T1728 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 778605 T1718 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 778606 T1754 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 778606 T1710 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 778606 T1744 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 778606 T1700 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 778606 T1722 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 778606 T1748 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 778606 T1704 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 778606 T1734 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 778606 T1746 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 778607 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 778609 T1756 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7180625b name:ZooKeeperConnection Watcher:127.0.0.1:56234/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 778609 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 778611 T1683 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 778615 T1683 oascc.SolrZkClient.makePath makePath: /live_nodes/node28
   [junit4]   2> 778622 T1716 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 778622 T1694 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 778624 T1742 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 778624 T1730 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 778626 T1754 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 778626 T1718 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 778626 T1738 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 778626 T1746 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 778627 T1714 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 778627 T1724 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 778627 T1722 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 778627 T1712 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 778627 T1704 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 778628 T1708 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 778628 T1706 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 778628 T1726 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 778628 T1702 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 778628 T1732 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 778628 T1736 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 778629 T1748 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 778629 T1744 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 778629 T1734 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 778629 T1728 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 778629 T1752 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 778629 T1750 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 778630 T1710 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 778630 T1740 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 778629 T1756 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 778630 T1700 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 778630 T1720 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 778630 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 778632 T1758 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@35772226 name:ZooKeeperConnection Watcher:127.0.0.1:56234/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 778633 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 778634 T1683 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 778637 T1683 oascc.SolrZkClient.makePath makePath: /live_nodes/node29
   [junit4]   2> 778644 T1722 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 778644 T1754 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 778648 T1702 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 778648 T1748 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 778650 T1710 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 778651 T1752 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 778651 T1706 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 778651 T1730 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 778651 T1738 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 778651 T1704 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 778652 T1736 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 778652 T1734 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 778652 T1712 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 778652 T1718 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 778652 T1756 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 778652 T1728 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 778652 T1740 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 778652 T1746 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 778652 T1700 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 778652 T1744 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 778652 T1732 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 778652 T1708 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 778653 T1714 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 778653 T1724 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 778653 T1750 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 778653 T1694 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 778653 T1683 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 778653 T1716 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 778653 T1758 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 778653 T1720 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 778653 T1726 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 778653 T1742 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 778663 T1760 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10cd99a7 name:ZooKeeperConnection Watcher:127.0.0.1:56234/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 778663 T1683 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 778666 T1683 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 778670 T1683 oascc.SolrZkClient.makePath makePath: /live_nodes/node30
   [junit4]   2> 778678 T1750 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 778678 T1708 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 778680 T1694 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 778680 T1714 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 778681 T1718 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 778682 T1760 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 778682 T1742 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 778683 T1746 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 778683 T1758 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 778684 T1720 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 778685 T1722 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 778685 T1734 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 778685 T1726 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]  

[...truncated too long message...]

ateHandler2.close closing DirectUpdateHandler2{commits=9,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=8,adds=8,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=29,cumulative_deletesById=3,cumulative_deletesByQuery=1,cumulative_errors=0,transaction_logs_total_size=24761,transaction_logs_total_number=9}
   [junit4]   2> 2826703 T7198 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 2826703 T7198 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 2826703 T7198 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 2826705 T7198 C2763 P59731 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@18a8609d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@a800b91),segFN=segments_a,generation=10}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@18a8609d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@a800b91),segFN=segments_b,generation=11}
   [junit4]   2> 2826706 T7198 C2763 P59731 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 11
   [junit4]   2> 2826708 T7198 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 2826708 T7198 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 2826708 T7198 oasc.CachingDirectoryFactory.closeCacheValue looking to close .\org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1392611447219/jetty6\ [CachedDir<<refCount=0;path=.\org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1392611447219/jetty6\;done=false>>]
   [junit4]   2> 2826708 T7198 oasc.CachingDirectoryFactory.close Closing directory: .\org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1392611447219/jetty6\
   [junit4]   2> 2826708 T7198 oasc.CachingDirectoryFactory.closeCacheValue looking to close .\org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1392611447219/jetty6\index [CachedDir<<refCount=0;path=.\org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1392611447219/jetty6\index;done=false>>]
   [junit4]   2> 2826708 T7198 oasc.CachingDirectoryFactory.close Closing directory: .\org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1392611447219/jetty6\index
   [junit4]   2> 2826708 T7198 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 2826708 T7198 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> NOTE: test params are: codec=Lucene42, sim=DefaultSimilarity, locale=vi_VN, timezone=America/Argentina/Tucuman
   [junit4]   2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.7.0_51 (64-bit)/cpus=2,threads=1,free=179552424,total=500695040
   [junit4]   2> NOTE: All tests run in this JVM: [TestCloudManagedSchema, OverseerRolesTest, XmlUpdateRequestHandlerTest, TestFastOutputStream, TestPostingsSolrHighlighter, TestFieldTypeCollectionResource, MBeansHandlerTest, DOMUtilTest, BasicFunctionalityTest, TestAnalyzeInfixSuggestions, TestRealTimeGet, DefaultValueUpdateProcessorTest, MoreLikeThisHandlerTest, TestBlendedInfixSuggestions, TestBadConfig, PathHierarchyTokenizerFactoryTest, TestXIncludeConfig, HdfsBasicDistributedZkTest, DistribCursorPagingTest, TestFoldingMultitermQuery, IndexSchemaTest, DeleteShardTest, TestUniqueKeyFieldResource, TestSerializedLuceneMatchVersion, MinimalSchemaTest, DirectUpdateHandlerTest, ShardSplitTest, URLClassifyProcessorTest, TestLMDirichletSimilarityFactory, TestNoOpRegenerator, MultiTermTest, TestStressUserVersions, TestOmitPositions, EchoParamsTest, ChangedSchemaMergeTest, BasicZkTest, TestCSVResponseWriter, TestValueSourceCache, TestAnalyzedSuggestions, TestIndexSearcher, TestStressRecovery, HdfsSyncSliceTest, ResponseLogComponentTest, FieldFacetExtrasTest, DocValuesMissingTest, TestSolrIndexConfig, TestRandomFaceting, AbstractAnalyticsFacetTest, TestSolrDeletionPolicy1, TestRangeQuery, NoCacheHeaderTest, TestCollationField, SOLR749Test, TestDefaultSimilarityFactory, SolrTestCaseJ4Test, SpellCheckComponentTest, SpellingQueryConverterTest, DirectSolrConnectionTest, TestRemoteStreaming, DirectUpdateHandlerOptimizeTest, SearchHandlerTest, TestStressVersions, TestCollapseQParserPlugin, DistributedTermsComponentTest, TestWriterPerf, TestArbitraryIndexDir, RemoteQueryErrorTest, PolyFieldTest, BlockCacheTest, TestCSVLoader, TestDFRSimilarityFactory, FieldMutatingUpdateProcessorTest, CopyFieldTest, NotRequiredUniqueKeyTest, LegacyHTMLStripCharFilterTest, TestJmxIntegration, HighlighterConfigTest, TestCoreContainer, SolrCoreTest, DateFieldTest, SpatialFilterTest, PluginInfoTest, ExpressionTest, DirectSolrSpellCheckerTest, OverseerTest, ConvertedLegacyTest, TestSolrQueryParser, TestSolrXmlPersistence, TestDynamicFieldResource, ExternalFileFieldSortTest, TestDistributedSearch, PeerSyncTest, TestDynamicFieldCollectionResource, StressHdfsTest, SolrCoreCheckLockOnStartupTest, TestUpdate, FileUtilsTest, TestFiltering, SolrRequestParserTest, CoreMergeIndexesAdminHandlerTest, IndexSchemaRuntimeFieldTest, TestQuerySenderListener, TimeZoneUtilsTest, TestFastLRUCache, TestCustomSort, TestSolrDeletionPolicy2, TestImplicitCoreProperties, TestOverriddenPrefixQueryForCustomFieldType, TestUtils, HdfsRecoveryZkTest, TestPseudoReturnFields, TestMultiCoreConfBootstrap, UniqFieldsUpdateProcessorFactoryTest, SuggesterFSTTest, TestFastWriter, TestDistributedGrouping, TestPhraseSuggestions, TestCodecSupport, SortByFunctionTest, TestDocSet, ChaosMonkeySafeLeaderTest, StandardRequestHandlerTest, StatelessScriptUpdateProcessorFactoryTest, SolrIndexSplitterTest, EnumFieldTest, SuggesterWFSTTest, DocValuesMultiTest, ShowFileRequestHandlerTest, TestSort, TermsComponentTest, QueryEqualityTest, ResourceLoaderTest, ChaosMonkeyNothingIsSafeTest, WordBreakSolrSpellCheckerTest, ScriptEngineTest, CustomCollectionTest, TestQueryUtils, LoggingHandlerTest, ZkSolrClientTest, TestElisionMultitermQuery, TestSolrCoreProperties, HdfsCollectionsAPIDistributedZkTest, ShardRoutingCustomTest, HdfsDirectoryTest, CachingDirectoryFactoryTest, PreAnalyzedUpdateProcessorTest, TestRecovery, HighlighterTest, TestConfig, SliceStateUpdateTest, FunctionTest, TestLuceneMatchVersion, DocumentBuilderTest, SolrXmlInZkTest, TestPartialUpdateDeduplication, OpenCloseCoreStressTest, TestFreeTextSuggestions, TestCopyFieldCollectionResource, UpdateParamsTest, PingRequestHandlerTest, TestRandomMergePolicy, ModifyConfFileTest, PrimUtilsTest, TestHighFrequencyDictionaryFactory, AddSchemaFieldsUpdateProcessorFactoryTest, PrimitiveFieldTypeTest, AssignTest, TestSolr4Spatial, BadComponentTest, TermVectorComponentTest, TestZkChroot, ClusterStateUpdateTest, StatsComponentTest, TestSchemaResource, SchemaVersionSpecificBehaviorTest, TestIBSimilarityFactory, TestSearchPerf, TestDefaultSearchFieldResource, BasicDistributedZkTest, BasicDistributedZk2Test, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest, SyncSliceTest, LeaderElectionIntegrationTest, ShardRoutingTest, RecoveryZkTest, FullSolrCloudDistribCmdsTest]
   [junit4] Completed in 102.79s, 1 test, 1 error <<< FAILURES!

[...truncated 549 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:453: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:433: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:39: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-targets.xml:37: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build.xml:189: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\common-build.xml:491: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:1268: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:901: There were test failures: 372 suites, 1604 tests, 2 errors, 1 failure, 40 ignored (12 assumptions)

Total time: 89 minutes 37 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_51 -XX:-UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_60-ea-b04) - Build # 3787 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/3787/
Java: 64bit/jdk1.7.0_60-ea-b04 -XX:-UseCompressedOops -XX:+UseG1GC

10 tests failed.
REGRESSION:  org.apache.solr.analytics.expression.ExpressionTest.multiplyTest

Error Message:
Caught exception in getStatResult, xPath = /response/lst[@name='stats']/lst[@name='mr']/double[@name='sum']  raw data: <?xml version="1.0" encoding="UTF-8"?> <response>  <lst name="responseHeader">   <int name="status">0</int>   <int name="QTime">20</int> </lst> <result name="response" numFound="100" start="0"> </result> <lst name="stats">   <lst name="stats_fields"/> </lst> <lst name="stats">   <lst name="csr">     <str name="cs1">this is the first</str>     <str name="cs2">this is the second</str>     <str name="cs3">this is the third</str>   </lst>   <lst name="dmr">     <str name="cma">+2MONTHS</str>     <str name="cme">+2YEARS</str>     <date name="dmma">1812-02-29T23:59:59Z</date>     <date name="dmme">1807-12-31T23:59:59Z</date>     <date name="max">1811-12-31T23:59:59Z</date>     <date name="median">1805-12-31T23:59:59Z</date>   </lst>   <lst name="cr">     <str name="ccmax">this is the secondstr9</str>     <str name="ccmin">this is the firststr0</str>     <str name="csmax">this is the second</str>     <str name="csmin">this is the first</str>     <str name="max">str9</str>     <str name="min">str0</str>   </lst>   <lst name="cnr">     <double name="c10">10.0</double>     <double name="c8">8.0</double>   </lst>   <lst name="cdr">     <date name="cd1">1800-12-31T23:59:59Z</date>     <date name="cd2">1804-06-30T23:59:59Z</date>     <str name="cs1">1800-12-31T23:59:59Z</str>     <str name="cs2">1804-06-30T23:59:59Z</str>   </lst>   <lst name="avr">     <double name="c">100.0</double>     <long name="count">100</long>     <null name="s"/>     <null name="sum"/>   </lst>   <lst name="rr">     <str name="max">str9</str>     <str name="min">str0</str>     <str name="rmax">9rts</str>     <str name="rmin">0rts</str>   </lst>   <lst name="pr">     <long name="count">100</long>     <null name="mc"/>     <null name="mean"/>     <null name="su"/>     <null name="sum"/>     <null name="unique"/>   </lst>   <lst name="ar">     <long name="count">100</long>     <null name="mcm"/>     <null name="mean"/>     <null name="median"/>     <null name="su"/>     <null name="sum"/>     <null name="unique"/>   </lst>   <lst name="dr">     <long name="count">100</long>     <null name="mc"/>     <null name="mean"/>     <null name="su"/>     <null name="sum"/>     <null name="unique"/>   </lst>   <lst name="nr">     <double name="c">-100.0</double>     <long name="count">100</long>     <null name="s"/>     <null name="sum"/>   </lst>   <lst name="mr">     <long name="count">100</long>     <null name="mcm"/>     <null name="mean"/>     <null name="median"/>     <null name="su"/>     <null name="sum"/>     <null name="unique"/>   </lst> </lst> </response> 

Stack Trace:
java.lang.AssertionError: Caught exception in getStatResult, xPath = /response/lst[@name='stats']/lst[@name='mr']/double[@name='sum'] 
raw data: <?xml version="1.0" encoding="UTF-8"?>
<response>

<lst name="responseHeader">
  <int name="status">0</int>
  <int name="QTime">20</int>
</lst>
<result name="response" numFound="100" start="0">
</result>
<lst name="stats">
  <lst name="stats_fields"/>
</lst>
<lst name="stats">
  <lst name="csr">
    <str name="cs1">this is the first</str>
    <str name="cs2">this is the second</str>
    <str name="cs3">this is the third</str>
  </lst>
  <lst name="dmr">
    <str name="cma">+2MONTHS</str>
    <str name="cme">+2YEARS</str>
    <date name="dmma">1812-02-29T23:59:59Z</date>
    <date name="dmme">1807-12-31T23:59:59Z</date>
    <date name="max">1811-12-31T23:59:59Z</date>
    <date name="median">1805-12-31T23:59:59Z</date>
  </lst>
  <lst name="cr">
    <str name="ccmax">this is the secondstr9</str>
    <str name="ccmin">this is the firststr0</str>
    <str name="csmax">this is the second</str>
    <str name="csmin">this is the first</str>
    <str name="max">str9</str>
    <str name="min">str0</str>
  </lst>
  <lst name="cnr">
    <double name="c10">10.0</double>
    <double name="c8">8.0</double>
  </lst>
  <lst name="cdr">
    <date name="cd1">1800-12-31T23:59:59Z</date>
    <date name="cd2">1804-06-30T23:59:59Z</date>
    <str name="cs1">1800-12-31T23:59:59Z</str>
    <str name="cs2">1804-06-30T23:59:59Z</str>
  </lst>
  <lst name="avr">
    <double name="c">100.0</double>
    <long name="count">100</long>
    <null name="s"/>
    <null name="sum"/>
  </lst>
  <lst name="rr">
    <str name="max">str9</str>
    <str name="min">str0</str>
    <str name="rmax">9rts</str>
    <str name="rmin">0rts</str>
  </lst>
  <lst name="pr">
    <long name="count">100</long>
    <null name="mc"/>
    <null name="mean"/>
    <null name="su"/>
    <null name="sum"/>
    <null name="unique"/>
  </lst>
  <lst name="ar">
    <long name="count">100</long>
    <null name="mcm"/>
    <null name="mean"/>
    <null name="median"/>
    <null name="su"/>
    <null name="sum"/>
    <null name="unique"/>
  </lst>
  <lst name="dr">
    <long name="count">100</long>
    <null name="mc"/>
    <null name="mean"/>
    <null name="su"/>
    <null name="sum"/>
    <null name="unique"/>
  </lst>
  <lst name="nr">
    <double name="c">-100.0</double>
    <long name="count">100</long>
    <null name="s"/>
    <null name="sum"/>
  </lst>
  <lst name="mr">
    <long name="count">100</long>
    <null name="mcm"/>
    <null name="mean"/>
    <null name="median"/>
    <null name="su"/>
    <null name="sum"/>
    <null name="unique"/>
  </lst>
</lst>
</response>

	at __randomizedtesting.SeedInfo.seed([A0EA1881981D4E25:CB6B06FFAD13784B]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.analytics.AbstractAnalyticsStatsTest.getStatResult(AbstractAnalyticsStatsTest.java:114)
	at org.apache.solr.analytics.expression.ExpressionTest.multiplyTest(ExpressionTest.java:92)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)


REGRESSION:  org.apache.solr.analytics.expression.ExpressionTest.divideTest

Error Message:
Caught exception in getStatResult, xPath = /response/lst[@name='stats']/lst[@name='dr']/double[@name='sum']  raw data: <?xml version="1.0" encoding="UTF-8"?> <response>  <lst name="responseHeader">   <int name="status">0</int>   <int name="QTime">20</int> </lst> <result name="response" numFound="100" start="0"> </result> <lst name="stats">   <lst name="stats_fields"/> </lst> <lst name="stats">   <lst name="csr">     <str name="cs1">this is the first</str>     <str name="cs2">this is the second</str>     <str name="cs3">this is the third</str>   </lst>   <lst name="dmr">     <str name="cma">+2MONTHS</str>     <str name="cme">+2YEARS</str>     <date name="dmma">1812-02-29T23:59:59Z</date>     <date name="dmme">1807-12-31T23:59:59Z</date>     <date name="max">1811-12-31T23:59:59Z</date>     <date name="median">1805-12-31T23:59:59Z</date>   </lst>   <lst name="cr">     <str name="ccmax">this is the secondstr9</str>     <str name="ccmin">this is the firststr0</str>     <str name="csmax">this is the second</str>     <str name="csmin">this is the first</str>     <str name="max">str9</str>     <str name="min">str0</str>   </lst>   <lst name="cnr">     <double name="c10">10.0</double>     <double name="c8">8.0</double>   </lst>   <lst name="cdr">     <date name="cd1">1800-12-31T23:59:59Z</date>     <date name="cd2">1804-06-30T23:59:59Z</date>     <str name="cs1">1800-12-31T23:59:59Z</str>     <str name="cs2">1804-06-30T23:59:59Z</str>   </lst>   <lst name="avr">     <double name="c">100.0</double>     <long name="count">100</long>     <null name="s"/>     <null name="sum"/>   </lst>   <lst name="rr">     <str name="max">str9</str>     <str name="min">str0</str>     <str name="rmax">9rts</str>     <str name="rmin">0rts</str>   </lst>   <lst name="pr">     <long name="count">100</long>     <null name="mc"/>     <null name="mean"/>     <null name="su"/>     <null name="sum"/>     <null name="unique"/>   </lst>   <lst name="ar">     <long name="count">100</long>     <null name="mcm"/>     <null name="mean"/>     <null name="median"/>     <null name="su"/>     <null name="sum"/>     <null name="unique"/>   </lst>   <lst name="dr">     <long name="count">100</long>     <null name="mc"/>     <null name="mean"/>     <null name="su"/>     <null name="sum"/>     <null name="unique"/>   </lst>   <lst name="nr">     <double name="c">-100.0</double>     <long name="count">100</long>     <null name="s"/>     <null name="sum"/>   </lst>   <lst name="mr">     <long name="count">100</long>     <null name="mcm"/>     <null name="mean"/>     <null name="median"/>     <null name="su"/>     <null name="sum"/>     <null name="unique"/>   </lst> </lst> </response> 

Stack Trace:
java.lang.AssertionError: Caught exception in getStatResult, xPath = /response/lst[@name='stats']/lst[@name='dr']/double[@name='sum'] 
raw data: <?xml version="1.0" encoding="UTF-8"?>
<response>

<lst name="responseHeader">
  <int name="status">0</int>
  <int name="QTime">20</int>
</lst>
<result name="response" numFound="100" start="0">
</result>
<lst name="stats">
  <lst name="stats_fields"/>
</lst>
<lst name="stats">
  <lst name="csr">
    <str name="cs1">this is the first</str>
    <str name="cs2">this is the second</str>
    <str name="cs3">this is the third</str>
  </lst>
  <lst name="dmr">
    <str name="cma">+2MONTHS</str>
    <str name="cme">+2YEARS</str>
    <date name="dmma">1812-02-29T23:59:59Z</date>
    <date name="dmme">1807-12-31T23:59:59Z</date>
    <date name="max">1811-12-31T23:59:59Z</date>
    <date name="median">1805-12-31T23:59:59Z</date>
  </lst>
  <lst name="cr">
    <str name="ccmax">this is the secondstr9</str>
    <str name="ccmin">this is the firststr0</str>
    <str name="csmax">this is the second</str>
    <str name="csmin">this is the first</str>
    <str name="max">str9</str>
    <str name="min">str0</str>
  </lst>
  <lst name="cnr">
    <double name="c10">10.0</double>
    <double name="c8">8.0</double>
  </lst>
  <lst name="cdr">
    <date name="cd1">1800-12-31T23:59:59Z</date>
    <date name="cd2">1804-06-30T23:59:59Z</date>
    <str name="cs1">1800-12-31T23:59:59Z</str>
    <str name="cs2">1804-06-30T23:59:59Z</str>
  </lst>
  <lst name="avr">
    <double name="c">100.0</double>
    <long name="count">100</long>
    <null name="s"/>
    <null name="sum"/>
  </lst>
  <lst name="rr">
    <str name="max">str9</str>
    <str name="min">str0</str>
    <str name="rmax">9rts</str>
    <str name="rmin">0rts</str>
  </lst>
  <lst name="pr">
    <long name="count">100</long>
    <null name="mc"/>
    <null name="mean"/>
    <null name="su"/>
    <null name="sum"/>
    <null name="unique"/>
  </lst>
  <lst name="ar">
    <long name="count">100</long>
    <null name="mcm"/>
    <null name="mean"/>
    <null name="median"/>
    <null name="su"/>
    <null name="sum"/>
    <null name="unique"/>
  </lst>
  <lst name="dr">
    <long name="count">100</long>
    <null name="mc"/>
    <null name="mean"/>
    <null name="su"/>
    <null name="sum"/>
    <null name="unique"/>
  </lst>
  <lst name="nr">
    <double name="c">-100.0</double>
    <long name="count">100</long>
    <null name="s"/>
    <null name="sum"/>
  </lst>
  <lst name="mr">
    <long name="count">100</long>
    <null name="mcm"/>
    <null name="mean"/>
    <null name="median"/>
    <null name="su"/>
    <null name="sum"/>
    <null name="unique"/>
  </lst>
</lst>
</response>

	at __randomizedtesting.SeedInfo.seed([A0EA1881981D4E25:F8E7F377BCDF2CF3]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.analytics.AbstractAnalyticsStatsTest.getStatResult(AbstractAnalyticsStatsTest.java:114)
	at org.apache.solr.analytics.expression.ExpressionTest.divideTest(ExpressionTest.java:106)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)


REGRESSION:  org.apache.solr.analytics.expression.ExpressionTest.addTest

Error Message:
Caught exception in getStatResult, xPath = /response/lst[@name='stats']/lst[@name='ar']/double[@name='sum']  raw data: <?xml version="1.0" encoding="UTF-8"?> <response>  <lst name="responseHeader">   <int name="status">0</int>   <int name="QTime">20</int> </lst> <result name="response" numFound="100" start="0"> </result> <lst name="stats">   <lst name="stats_fields"/> </lst> <lst name="stats">   <lst name="csr">     <str name="cs1">this is the first</str>     <str name="cs2">this is the second</str>     <str name="cs3">this is the third</str>   </lst>   <lst name="dmr">     <str name="cma">+2MONTHS</str>     <str name="cme">+2YEARS</str>     <date name="dmma">1812-02-29T23:59:59Z</date>     <date name="dmme">1807-12-31T23:59:59Z</date>     <date name="max">1811-12-31T23:59:59Z</date>     <date name="median">1805-12-31T23:59:59Z</date>   </lst>   <lst name="cr">     <str name="ccmax">this is the secondstr9</str>     <str name="ccmin">this is the firststr0</str>     <str name="csmax">this is the second</str>     <str name="csmin">this is the first</str>     <str name="max">str9</str>     <str name="min">str0</str>   </lst>   <lst name="cnr">     <double name="c10">10.0</double>     <double name="c8">8.0</double>   </lst>   <lst name="cdr">     <date name="cd1">1800-12-31T23:59:59Z</date>     <date name="cd2">1804-06-30T23:59:59Z</date>     <str name="cs1">1800-12-31T23:59:59Z</str>     <str name="cs2">1804-06-30T23:59:59Z</str>   </lst>   <lst name="avr">     <double name="c">100.0</double>     <long name="count">100</long>     <null name="s"/>     <null name="sum"/>   </lst>   <lst name="rr">     <str name="max">str9</str>     <str name="min">str0</str>     <str name="rmax">9rts</str>     <str name="rmin">0rts</str>   </lst>   <lst name="pr">     <long name="count">100</long>     <null name="mc"/>     <null name="mean"/>     <null name="su"/>     <null name="sum"/>     <null name="unique"/>   </lst>   <lst name="ar">     <long name="count">100</long>     <null name="mcm"/>     <null name="mean"/>     <null name="median"/>     <null name="su"/>     <null name="sum"/>     <null name="unique"/>   </lst>   <lst name="dr">     <long name="count">100</long>     <null name="mc"/>     <null name="mean"/>     <null name="su"/>     <null name="sum"/>     <null name="unique"/>   </lst>   <lst name="nr">     <double name="c">-100.0</double>     <long name="count">100</long>     <null name="s"/>     <null name="sum"/>   </lst>   <lst name="mr">     <long name="count">100</long>     <null name="mcm"/>     <null name="mean"/>     <null name="median"/>     <null name="su"/>     <null name="sum"/>     <null name="unique"/>   </lst> </lst> </response> 

Stack Trace:
java.lang.AssertionError: Caught exception in getStatResult, xPath = /response/lst[@name='stats']/lst[@name='ar']/double[@name='sum'] 
raw data: <?xml version="1.0" encoding="UTF-8"?>
<response>

<lst name="responseHeader">
  <int name="status">0</int>
  <int name="QTime">20</int>
</lst>
<result name="response" numFound="100" start="0">
</result>
<lst name="stats">
  <lst name="stats_fields"/>
</lst>
<lst name="stats">
  <lst name="csr">
    <str name="cs1">this is the first</str>
    <str name="cs2">this is the second</str>
    <str name="cs3">this is the third</str>
  </lst>
  <lst name="dmr">
    <str name="cma">+2MONTHS</str>
    <str name="cme">+2YEARS</str>
    <date name="dmma">1812-02-29T23:59:59Z</date>
    <date name="dmme">1807-12-31T23:59:59Z</date>
    <date name="max">1811-12-31T23:59:59Z</date>
    <date name="median">1805-12-31T23:59:59Z</date>
  </lst>
  <lst name="cr">
    <str name="ccmax">this is the secondstr9</str>
    <str name="ccmin">this is the firststr0</str>
    <str name="csmax">this is the second</str>
    <str name="csmin">this is the first</str>
    <str name="max">str9</str>
    <str name="min">str0</str>
  </lst>
  <lst name="cnr">
    <double name="c10">10.0</double>
    <double name="c8">8.0</double>
  </lst>
  <lst name="cdr">
    <date name="cd1">1800-12-31T23:59:59Z</date>
    <date name="cd2">1804-06-30T23:59:59Z</date>
    <str name="cs1">1800-12-31T23:59:59Z</str>
    <str name="cs2">1804-06-30T23:59:59Z</str>
  </lst>
  <lst name="avr">
    <double name="c">100.0</double>
    <long name="count">100</long>
    <null name="s"/>
    <null name="sum"/>
  </lst>
  <lst name="rr">
    <str name="max">str9</str>
    <str name="min">str0</str>
    <str name="rmax">9rts</str>
    <str name="rmin">0rts</str>
  </lst>
  <lst name="pr">
    <long name="count">100</long>
    <null name="mc"/>
    <null name="mean"/>
    <null name="su"/>
    <null name="sum"/>
    <null name="unique"/>
  </lst>
  <lst name="ar">
    <long name="count">100</long>
    <null name="mcm"/>
    <null name="mean"/>
    <null name="median"/>
    <null name="su"/>
    <null name="sum"/>
    <null name="unique"/>
  </lst>
  <lst name="dr">
    <long name="count">100</long>
    <null name="mc"/>
    <null name="mean"/>
    <null name="su"/>
    <null name="sum"/>
    <null name="unique"/>
  </lst>
  <lst name="nr">
    <double name="c">-100.0</double>
    <long name="count">100</long>
    <null name="s"/>
    <null name="sum"/>
  </lst>
  <lst name="mr">
    <long name="count">100</long>
    <null name="mcm"/>
    <null name="mean"/>
    <null name="median"/>
    <null name="su"/>
    <null name="sum"/>
    <null name="unique"/>
  </lst>
</lst>
</response>

	at __randomizedtesting.SeedInfo.seed([A0EA1881981D4E25:56C6B55FD9D24962]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.analytics.AbstractAnalyticsStatsTest.getStatResult(AbstractAnalyticsStatsTest.java:114)
	at org.apache.solr.analytics.expression.ExpressionTest.addTest(ExpressionTest.java:78)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)


REGRESSION:  org.apache.solr.analytics.expression.ExpressionTest.negateTest

Error Message:
Caught exception in getStatResult, xPath = /response/lst[@name='stats']/lst[@name='nr']/double[@name='sum']  raw data: <?xml version="1.0" encoding="UTF-8"?> <response>  <lst name="responseHeader">   <int name="status">0</int>   <int name="QTime">20</int> </lst> <result name="response" numFound="100" start="0"> </result> <lst name="stats">   <lst name="stats_fields"/> </lst> <lst name="stats">   <lst name="csr">     <str name="cs1">this is the first</str>     <str name="cs2">this is the second</str>     <str name="cs3">this is the third</str>   </lst>   <lst name="dmr">     <str name="cma">+2MONTHS</str>     <str name="cme">+2YEARS</str>     <date name="dmma">1812-02-29T23:59:59Z</date>     <date name="dmme">1807-12-31T23:59:59Z</date>     <date name="max">1811-12-31T23:59:59Z</date>     <date name="median">1805-12-31T23:59:59Z</date>   </lst>   <lst name="cr">     <str name="ccmax">this is the secondstr9</str>     <str name="ccmin">this is the firststr0</str>     <str name="csmax">this is the second</str>     <str name="csmin">this is the first</str>     <str name="max">str9</str>     <str name="min">str0</str>   </lst>   <lst name="cnr">     <double name="c10">10.0</double>     <double name="c8">8.0</double>   </lst>   <lst name="cdr">     <date name="cd1">1800-12-31T23:59:59Z</date>     <date name="cd2">1804-06-30T23:59:59Z</date>     <str name="cs1">1800-12-31T23:59:59Z</str>     <str name="cs2">1804-06-30T23:59:59Z</str>   </lst>   <lst name="avr">     <double name="c">100.0</double>     <long name="count">100</long>     <null name="s"/>     <null name="sum"/>   </lst>   <lst name="rr">     <str name="max">str9</str>     <str name="min">str0</str>     <str name="rmax">9rts</str>     <str name="rmin">0rts</str>   </lst>   <lst name="pr">     <long name="count">100</long>     <null name="mc"/>     <null name="mean"/>     <null name="su"/>     <null name="sum"/>     <null name="unique"/>   </lst>   <lst name="ar">     <long name="count">100</long>     <null name="mcm"/>     <null name="mean"/>     <null name="median"/>     <null name="su"/>     <null name="sum"/>     <null name="unique"/>   </lst>   <lst name="dr">     <long name="count">100</long>     <null name="mc"/>     <null name="mean"/>     <null name="su"/>     <null name="sum"/>     <null name="unique"/>   </lst>   <lst name="nr">     <double name="c">-100.0</double>     <long name="count">100</long>     <null name="s"/>     <null name="sum"/>   </lst>   <lst name="mr">     <long name="count">100</long>     <null name="mcm"/>     <null name="mean"/>     <null name="median"/>     <null name="su"/>     <null name="sum"/>     <null name="unique"/>   </lst> </lst> </response> 

Stack Trace:
java.lang.AssertionError: Caught exception in getStatResult, xPath = /response/lst[@name='stats']/lst[@name='nr']/double[@name='sum'] 
raw data: <?xml version="1.0" encoding="UTF-8"?>
<response>

<lst name="responseHeader">
  <int name="status">0</int>
  <int name="QTime">20</int>
</lst>
<result name="response" numFound="100" start="0">
</result>
<lst name="stats">
  <lst name="stats_fields"/>
</lst>
<lst name="stats">
  <lst name="csr">
    <str name="cs1">this is the first</str>
    <str name="cs2">this is the second</str>
    <str name="cs3">this is the third</str>
  </lst>
  <lst name="dmr">
    <str name="cma">+2MONTHS</str>
    <str name="cme">+2YEARS</str>
    <date name="dmma">1812-02-29T23:59:59Z</date>
    <date name="dmme">1807-12-31T23:59:59Z</date>
    <date name="max">1811-12-31T23:59:59Z</date>
    <date name="median">1805-12-31T23:59:59Z</date>
  </lst>
  <lst name="cr">
    <str name="ccmax">this is the secondstr9</str>
    <str name="ccmin">this is the firststr0</str>
    <str name="csmax">this is the second</str>
    <str name="csmin">this is the first</str>
    <str name="max">str9</str>
    <str name="min">str0</str>
  </lst>
  <lst name="cnr">
    <double name="c10">10.0</double>
    <double name="c8">8.0</double>
  </lst>
  <lst name="cdr">
    <date name="cd1">1800-12-31T23:59:59Z</date>
    <date name="cd2">1804-06-30T23:59:59Z</date>
    <str name="cs1">1800-12-31T23:59:59Z</str>
    <str name="cs2">1804-06-30T23:59:59Z</str>
  </lst>
  <lst name="avr">
    <double name="c">100.0</double>
    <long name="count">100</long>
    <null name="s"/>
    <null name="sum"/>
  </lst>
  <lst name="rr">
    <str name="max">str9</str>
    <str name="min">str0</str>
    <str name="rmax">9rts</str>
    <str name="rmin">0rts</str>
  </lst>
  <lst name="pr">
    <long name="count">100</long>
    <null name="mc"/>
    <null name="mean"/>
    <null name="su"/>
    <null name="sum"/>
    <null name="unique"/>
  </lst>
  <lst name="ar">
    <long name="count">100</long>
    <null name="mcm"/>
    <null name="mean"/>
    <null name="median"/>
    <null name="su"/>
    <null name="sum"/>
    <null name="unique"/>
  </lst>
  <lst name="dr">
    <long name="count">100</long>
    <null name="mc"/>
    <null name="mean"/>
    <null name="su"/>
    <null name="sum"/>
    <null name="unique"/>
  </lst>
  <lst name="nr">
    <double name="c">-100.0</double>
    <long name="count">100</long>
    <null name="s"/>
    <null name="sum"/>
  </lst>
  <lst name="mr">
    <long name="count">100</long>
    <null name="mcm"/>
    <null name="mean"/>
    <null name="median"/>
    <null name="su"/>
    <null name="sum"/>
    <null name="unique"/>
  </lst>
</lst>
</response>

	at __randomizedtesting.SeedInfo.seed([A0EA1881981D4E25:A8E270243C301E95]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.analytics.AbstractAnalyticsStatsTest.getStatResult(AbstractAnalyticsStatsTest.java:114)
	at org.apache.solr.analytics.expression.ExpressionTest.negateTest(ExpressionTest.java:132)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)


REGRESSION:  org.apache.solr.analytics.expression.ExpressionTest.absoluteValueTest

Error Message:
Caught exception in getStatResult, xPath = /response/lst[@name='stats']/lst[@name='avr']/double[@name='sum']  raw data: <?xml version="1.0" encoding="UTF-8"?> <response>  <lst name="responseHeader">   <int name="status">0</int>   <int name="QTime">20</int> </lst> <result name="response" numFound="100" start="0"> </result> <lst name="stats">   <lst name="stats_fields"/> </lst> <lst name="stats">   <lst name="csr">     <str name="cs1">this is the first</str>     <str name="cs2">this is the second</str>     <str name="cs3">this is the third</str>   </lst>   <lst name="dmr">     <str name="cma">+2MONTHS</str>     <str name="cme">+2YEARS</str>     <date name="dmma">1812-02-29T23:59:59Z</date>     <date name="dmme">1807-12-31T23:59:59Z</date>     <date name="max">1811-12-31T23:59:59Z</date>     <date name="median">1805-12-31T23:59:59Z</date>   </lst>   <lst name="cr">     <str name="ccmax">this is the secondstr9</str>     <str name="ccmin">this is the firststr0</str>     <str name="csmax">this is the second</str>     <str name="csmin">this is the first</str>     <str name="max">str9</str>     <str name="min">str0</str>   </lst>   <lst name="cnr">     <double name="c10">10.0</double>     <double name="c8">8.0</double>   </lst>   <lst name="cdr">     <date name="cd1">1800-12-31T23:59:59Z</date>     <date name="cd2">1804-06-30T23:59:59Z</date>     <str name="cs1">1800-12-31T23:59:59Z</str>     <str name="cs2">1804-06-30T23:59:59Z</str>   </lst>   <lst name="avr">     <double name="c">100.0</double>     <long name="count">100</long>     <null name="s"/>     <null name="sum"/>   </lst>   <lst name="rr">     <str name="max">str9</str>     <str name="min">str0</str>     <str name="rmax">9rts</str>     <str name="rmin">0rts</str>   </lst>   <lst name="pr">     <long name="count">100</long>     <null name="mc"/>     <null name="mean"/>     <null name="su"/>     <null name="sum"/>     <null name="unique"/>   </lst>   <lst name="ar">     <long name="count">100</long>     <null name="mcm"/>     <null name="mean"/>     <null name="median"/>     <null name="su"/>     <null name="sum"/>     <null name="unique"/>   </lst>   <lst name="dr">     <long name="count">100</long>     <null name="mc"/>     <null name="mean"/>     <null name="su"/>     <null name="sum"/>     <null name="unique"/>   </lst>   <lst name="nr">     <double name="c">-100.0</double>     <long name="count">100</long>     <null name="s"/>     <null name="sum"/>   </lst>   <lst name="mr">     <long name="count">100</long>     <null name="mcm"/>     <null name="mean"/>     <null name="median"/>     <null name="su"/>     <null name="sum"/>     <null name="unique"/>   </lst> </lst> </response> 

Stack Trace:
java.lang.AssertionError: Caught exception in getStatResult, xPath = /response/lst[@name='stats']/lst[@name='avr']/double[@name='sum'] 
raw data: <?xml version="1.0" encoding="UTF-8"?>
<response>

<lst name="responseHeader">
  <int name="status">0</int>
  <int name="QTime">20</int>
</lst>
<result name="response" numFound="100" start="0">
</result>
<lst name="stats">
  <lst name="stats_fields"/>
</lst>
<lst name="stats">
  <lst name="csr">
    <str name="cs1">this is the first</str>
    <str name="cs2">this is the second</str>
    <str name="cs3">this is the third</str>
  </lst>
  <lst name="dmr">
    <str name="cma">+2MONTHS</str>
    <str name="cme">+2YEARS</str>
    <date name="dmma">1812-02-29T23:59:59Z</date>
    <date name="dmme">1807-12-31T23:59:59Z</date>
    <date name="max">1811-12-31T23:59:59Z</date>
    <date name="median">1805-12-31T23:59:59Z</date>
  </lst>
  <lst name="cr">
    <str name="ccmax">this is the secondstr9</str>
    <str name="ccmin">this is the firststr0</str>
    <str name="csmax">this is the second</str>
    <str name="csmin">this is the first</str>
    <str name="max">str9</str>
    <str name="min">str0</str>
  </lst>
  <lst name="cnr">
    <double name="c10">10.0</double>
    <double name="c8">8.0</double>
  </lst>
  <lst name="cdr">
    <date name="cd1">1800-12-31T23:59:59Z</date>
    <date name="cd2">1804-06-30T23:59:59Z</date>
    <str name="cs1">1800-12-31T23:59:59Z</str>
    <str name="cs2">1804-06-30T23:59:59Z</str>
  </lst>
  <lst name="avr">
    <double name="c">100.0</double>
    <long name="count">100</long>
    <null name="s"/>
    <null name="sum"/>
  </lst>
  <lst name="rr">
    <str name="max">str9</str>
    <str name="min">str0</str>
    <str name="rmax">9rts</str>
    <str name="rmin">0rts</str>
  </lst>
  <lst name="pr">
    <long name="count">100</long>
    <null name="mc"/>
    <null name="mean"/>
    <null name="su"/>
    <null name="sum"/>
    <null name="unique"/>
  </lst>
  <lst name="ar">
    <long name="count">100</long>
    <null name="mcm"/>
    <null name="mean"/>
    <null name="median"/>
    <null name="su"/>
    <null name="sum"/>
    <null name="unique"/>
  </lst>
  <lst name="dr">
    <long name="count">100</long>
    <null name="mc"/>
    <null name="mean"/>
    <null name="su"/>
    <null name="sum"/>
    <null name="unique"/>
  </lst>
  <lst name="nr">
    <double name="c">-100.0</double>
    <long name="count">100</long>
    <null name="s"/>
    <null name="sum"/>
  </lst>
  <lst name="mr">
    <long name="count">100</long>
    <null name="mcm"/>
    <null name="mean"/>
    <null name="median"/>
    <null name="su"/>
    <null name="sum"/>
    <null name="unique"/>
  </lst>
</lst>
</response>

	at __randomizedtesting.SeedInfo.seed([A0EA1881981D4E25:8146389455801643]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.analytics.AbstractAnalyticsStatsTest.getStatResult(AbstractAnalyticsStatsTest.java:114)
	at org.apache.solr.analytics.expression.ExpressionTest.absoluteValueTest(ExpressionTest.java:143)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)


REGRESSION:  org.apache.solr.analytics.expression.ExpressionTest.powerTest

Error Message:
Caught exception in getStatResult, xPath = /response/lst[@name='stats']/lst[@name='pr']/double[@name='sum']  raw data: <?xml version="1.0" encoding="UTF-8"?> <response>  <lst name="responseHeader">   <int name="status">0</int>   <int name="QTime">20</int> </lst> <result name="response" numFound="100" start="0"> </result> <lst name="stats">   <lst name="stats_fields"/> </lst> <lst name="stats">   <lst name="csr">     <str name="cs1">this is the first</str>     <str name="cs2">this is the second</str>     <str name="cs3">this is the third</str>   </lst>   <lst name="dmr">     <str name="cma">+2MONTHS</str>     <str name="cme">+2YEARS</str>     <date name="dmma">1812-02-29T23:59:59Z</date>     <date name="dmme">1807-12-31T23:59:59Z</date>     <date name="max">1811-12-31T23:59:59Z</date>     <date name="median">1805-12-31T23:59:59Z</date>   </lst>   <lst name="cr">     <str name="ccmax">this is the secondstr9</str>     <str name="ccmin">this is the firststr0</str>     <str name="csmax">this is the second</str>     <str name="csmin">this is the first</str>     <str name="max">str9</str>     <str name="min">str0</str>   </lst>   <lst name="cnr">     <double name="c10">10.0</double>     <double name="c8">8.0</double>   </lst>   <lst name="cdr">     <date name="cd1">1800-12-31T23:59:59Z</date>     <date name="cd2">1804-06-30T23:59:59Z</date>     <str name="cs1">1800-12-31T23:59:59Z</str>     <str name="cs2">1804-06-30T23:59:59Z</str>   </lst>   <lst name="avr">     <double name="c">100.0</double>     <long name="count">100</long>     <null name="s"/>     <null name="sum"/>   </lst>   <lst name="rr">     <str name="max">str9</str>     <str name="min">str0</str>     <str name="rmax">9rts</str>     <str name="rmin">0rts</str>   </lst>   <lst name="pr">     <long name="count">100</long>     <null name="mc"/>     <null name="mean"/>     <null name="su"/>     <null name="sum"/>     <null name="unique"/>   </lst>   <lst name="ar">     <long name="count">100</long>     <null name="mcm"/>     <null name="mean"/>     <null name="median"/>     <null name="su"/>     <null name="sum"/>     <null name="unique"/>   </lst>   <lst name="dr">     <long name="count">100</long>     <null name="mc"/>     <null name="mean"/>     <null name="su"/>     <null name="sum"/>     <null name="unique"/>   </lst>   <lst name="nr">     <double name="c">-100.0</double>     <long name="count">100</long>     <null name="s"/>     <null name="sum"/>   </lst>   <lst name="mr">     <long name="count">100</long>     <null name="mcm"/>     <null name="mean"/>     <null name="median"/>     <null name="su"/>     <null name="sum"/>     <null name="unique"/>   </lst> </lst> </response> 

Stack Trace:
java.lang.AssertionError: Caught exception in getStatResult, xPath = /response/lst[@name='stats']/lst[@name='pr']/double[@name='sum'] 
raw data: <?xml version="1.0" encoding="UTF-8"?>
<response>

<lst name="responseHeader">
  <int name="status">0</int>
  <int name="QTime">20</int>
</lst>
<result name="response" numFound="100" start="0">
</result>
<lst name="stats">
  <lst name="stats_fields"/>
</lst>
<lst name="stats">
  <lst name="csr">
    <str name="cs1">this is the first</str>
    <str name="cs2">this is the second</str>
    <str name="cs3">this is the third</str>
  </lst>
  <lst name="dmr">
    <str name="cma">+2MONTHS</str>
    <str name="cme">+2YEARS</str>
    <date name="dmma">1812-02-29T23:59:59Z</date>
    <date name="dmme">1807-12-31T23:59:59Z</date>
    <date name="max">1811-12-31T23:59:59Z</date>
    <date name="median">1805-12-31T23:59:59Z</date>
  </lst>
  <lst name="cr">
    <str name="ccmax">this is the secondstr9</str>
    <str name="ccmin">this is the firststr0</str>
    <str name="csmax">this is the second</str>
    <str name="csmin">this is the first</str>
    <str name="max">str9</str>
    <str name="min">str0</str>
  </lst>
  <lst name="cnr">
    <double name="c10">10.0</double>
    <double name="c8">8.0</double>
  </lst>
  <lst name="cdr">
    <date name="cd1">1800-12-31T23:59:59Z</date>
    <date name="cd2">1804-06-30T23:59:59Z</date>
    <str name="cs1">1800-12-31T23:59:59Z</str>
    <str name="cs2">1804-06-30T23:59:59Z</str>
  </lst>
  <lst name="avr">
    <double name="c">100.0</double>
    <long name="count">100</long>
    <null name="s"/>
    <null name="sum"/>
  </lst>
  <lst name="rr">
    <str name="max">str9</str>
    <str name="min">str0</str>
    <str name="rmax">9rts</str>
    <str name="rmin">0rts</str>
  </lst>
  <lst name="pr">
    <long name="count">100</long>
    <null name="mc"/>
    <null name="mean"/>
    <null name="su"/>
    <null name="sum"/>
    <null name="unique"/>
  </lst>
  <lst name="ar">
    <long name="count">100</long>
    <null name="mcm"/>
    <null name="mean"/>
    <null name="median"/>
    <null name="su"/>
    <null name="sum"/>
    <null name="unique"/>
  </lst>
  <lst name="dr">
    <long name="count">100</long>
    <null name="mc"/>
    <null name="mean"/>
    <null name="su"/>
    <null name="sum"/>
    <null name="unique"/>
  </lst>
  <lst name="nr">
    <double name="c">-100.0</double>
    <long name="count">100</long>
    <null name="s"/>
    <null name="sum"/>
  </lst>
  <lst name="mr">
    <long name="count">100</long>
    <null name="mcm"/>
    <null name="mean"/>
    <null name="median"/>
    <null name="su"/>
    <null name="sum"/>
    <null name="unique"/>
  </lst>
</lst>
</response>

	at __randomizedtesting.SeedInfo.seed([A0EA1881981D4E25:CE58AF82700A10F4]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.analytics.AbstractAnalyticsStatsTest.getStatResult(AbstractAnalyticsStatsTest.java:114)
	at org.apache.solr.analytics.expression.ExpressionTest.powerTest(ExpressionTest.java:119)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.solr.TestDistributedGrouping.testDistribSearch

Error Message:
Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 Server Error</title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /er_x/select. Reason: <pre>    Server Error</pre></p> <hr /><i><small>Powered by Jetty://</small></i>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 Server Error</title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /er_x/select. Reason:
<pre>    Server Error</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
</body>
</html>

	at __randomizedtesting.SeedInfo.seed([A0EA1881981D4E25:210C9699EF422E19]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:459)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:90)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.BaseDistributedSearchTestCase.queryServer(BaseDistributedSearchTestCase.java:515)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:563)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:545)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:524)
	at org.apache.solr.TestDistributedGrouping.doTest(TestDistributedGrouping.java:156)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:876)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.solr.TestDistributedSearch.testDistribSearch

Error Message:
Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 Server Error</title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /select. Reason: <pre>    Server Error</pre></p> <hr /><i><small>Powered by Jetty://</small></i>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 Server Error</title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /select. Reason:
<pre>    Server Error</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
</body>
</html>

	at __randomizedtesting.SeedInfo.seed([A0EA1881981D4E25:210C9699EF422E19]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:459)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:90)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.BaseDistributedSearchTestCase.queryServer(BaseDistributedSearchTestCase.java:515)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:563)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:545)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:524)
	at org.apache.solr.TestDistributedSearch.doTest(TestDistributedSearch.java:177)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:876)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.solr.cloud.BasicDistributedZk2Test.testDistribSearch

Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:55731/collection1, http://127.0.0.1:55722/collection1, http://127.0.0.1:55741/collection1, http://127.0.0.1:55713/collection1]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:55731/collection1, http://127.0.0.1:55722/collection1, http://127.0.0.1:55741/collection1, http://127.0.0.1:55713/collection1]
	at __randomizedtesting.SeedInfo.seed([A0EA1881981D4E25:210C9699EF422E19]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:362)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:635)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:90)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.queryServer(AbstractFullDistribZkTestBase.java:1356)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:563)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:545)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:524)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkQueries(AbstractFullDistribZkTestBase.java:797)
	at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:101)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 Server Error</title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /collection1/select. Reason:
<pre>    Server Error</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
</body>
</html>

	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:459)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:293)
	... 50 more


FAILED:  org.apache.solr.cloud.BasicDistributedZkTest.testDistribSearch

Error Message:
No live SolrServers available to handle this request:[https://127.0.0.1:55039/a_ohq/u/collection1, https://127.0.0.1:55030/a_ohq/u/collection1, https://127.0.0.1:55048/a_ohq/u/collection1, https://127.0.0.1:55058/a_ohq/u/collection1]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:55039/a_ohq/u/collection1, https://127.0.0.1:55030/a_ohq/u/collection1, https://127.0.0.1:55048/a_ohq/u/collection1, https://127.0.0.1:55058/a_ohq/u/collection1]
	at __randomizedtesting.SeedInfo.seed([A0EA1881981D4E25:210C9699EF422E19]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:362)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:635)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:90)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.queryServer(AbstractFullDistribZkTestBase.java:1356)
	at org.apache.solr.cloud.BasicDistributedZkTest.queryServer(BasicDistributedZkTest.java:1171)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:563)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:545)
	at org.apache.solr.cloud.BasicDistributedZkTest.doTest(BasicDistributedZkTest.java:264)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 Server Error</title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /a_ohq/u/collection1/select. Reason:
<pre>    Server Error</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
</body>
</html>

	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:459)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:293)
	... 49 more




Build Log:
[...truncated 10160 lines...]
   [junit4] Suite: org.apache.solr.TestDistributedSearch
   [junit4]   2> 617734 T2208 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 617735 T2208 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 617738 T2208 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 617742 T2208 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 617746 T2208 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50007
   [junit4]   2> 617747 T2208 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 617748 T2208 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 617748 T2208 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr
   [junit4]   2> 617748 T2208 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\'
   [junit4]   2> 617780 T2208 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core

[...truncated too long message...]

[junit4]   2> 2655718 T9211 oasc.CachingDirectoryFactory.close Closing directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\solrtest-ExpressionTest-1392584667099\index
   [junit4]   2> NOTE: test params are: codec=Lucene46: {date_dtd=PostingsFormat(name=SimpleText), long_ld=PostingsFormat(name=SimpleText), int_id=PostingsFormat(name=SimpleText), float_fd=PostingsFormat(name=Lucene41WithOrds), id=PostingsFormat(name=SimpleText), double_dd=PostingsFormat(name=MockRandom), string_sd=PostingsFormat(name=SimpleText)}, docValues:{date_dtd=DocValuesFormat(name=Disk), stringdvm=DocValuesFormat(name=Lucene45), int_id=DocValuesFormat(name=Disk), long_ld=DocValuesFormat(name=Disk), float_fd=DocValuesFormat(name=Memory), longdv=DocValuesFormat(name=Lucene45), datedv=DocValuesFormat(name=SimpleText), floatdv=DocValuesFormat(name=Lucene45), doubledv=DocValuesFormat(name=Disk), stringdv=DocValuesFormat(name=Disk), double_dd=DocValuesFormat(name=Lucene45), intdv=DocValuesFormat(name=Memory), string_sd=DocValuesFormat(name=Disk)}, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=zh_TW, timezone=Europe/Moscow
   [junit4]   2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.7.0_60-ea (64-bit)/cpus=2,threads=1,free=110175032,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [SimplePostToolTest, TestFaceting, QueryFacetTest, HdfsBasicDistributedZk2Test, HdfsCollectionsAPIDistributedZkTest, TestStandardQParsers, TestFunctionQuery, QueryEqualityTest, BadCopyFieldTest, TestSolrXMLSerializer, TestLuceneMatchVersion, TestIBSimilarityFactory, TestPHPSerializedResponseWriter, LukeRequestHandlerTest, FieldMutatingUpdateProcessorTest, DistribCursorPagingTest, TestManagedSchemaFieldResource, HdfsLockFactoryTest, TestReversedWildcardFilterFactory, RequiredFieldsTest, TestSweetSpotSimilarityFactory, IndexBasedSpellCheckerTest, TestRangeQuery, TestSolr4Spatial, TestBlendedInfixSuggestions, ResourceLoaderTest, FullSolrCloudDistribCmdsTest, TestFieldTypeResource, OverseerTest, TestSerializedLuceneMatchVersion, TestDistributedMissingSort, TestDefaultSimilarityFactory, InfoHandlerTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestUpdate, SolrInfoMBeanTest, TestDistribDocBasedVersion, TestStressUserVersions, WordBreakSolrSpellCheckerTest, ModifyConfFileTest, TestDynamicFieldCollectionResource, DocumentBuilderTest, CacheHeaderTest, SpellPossibilityIteratorTest, TestRecoveryHdfs, SpatialFilterTest, OpenExchangeRatesOrgProviderTest, TestCSVResponseWriter, TestSchemaResource, TestRealTimeGet, TestAnalyzeInfixSuggestions, EchoParamsTest, CoreMergeIndexesAdminHandlerTest, TestReload, DateMathParserTest, TestSchemaNameResource, PrimitiveFieldTypeTest, SolrRequestParserTest, TestSolrIndexConfig, RangeFacetTest, TestAddFieldRealTimeGet, SolrCmdDistributorTest, BadComponentTest, TestTrie, HdfsBasicDistributedZkTest, DistributedSpellCheckComponentTest, LeaderElectionIntegrationTest, TestStressLucene, DateFieldTest, TestPartialUpdateDeduplication, TestDocBasedVersionConstraints, IndexSchemaRuntimeFieldTest, CachingDirectoryFactoryTest, TestFastWriter, XsltUpdateRequestHandlerTest, TestDistributedSearch, SoftAutoCommitTest, FastVectorHighlighterTest, SyncSliceTest, TestSchemaSimilarityResource, AddBlockUpdateTest, TestConfig, TestWriterPerf, StatelessScriptUpdateProcessorFactoryTest, DirectSolrSpellCheckerTest, PrimUtilsTest, TestSystemIdResolver, TestDynamicFieldResource, FunctionTest, TestUniqueKeyFieldResource, TestAnalyzedSuggestions, TestCSVLoader, DistributedTermsComponentTest, BJQParserTest, SpellCheckCollatorTest, FieldAnalysisRequestHandlerTest, MultiTermTest, SystemInfoHandlerTest, PluginInfoTest, MigrateRouteKeyTest, JsonLoaderTest, ClusterStateUpdateTest, TestCustomSort, DeleteShardTest, SearchHandlerTest, TestMaxScoreQueryParser, TestFieldCollectionResource, TestHighFrequencyDictionaryFactory, TestLazyCores, TestFiltering, PolyFieldTest, TestNoOpRegenerator, SuggesterWFSTTest, TestSurroundQueryParser, ResponseLogComponentTest, TestMergePolicyConfig, TermVectorComponentTest, DistributedSuggestComponentTest, CopyFieldTest, TestShardHandlerFactory, TestNumberUtils, SolrIndexSplitterTest, ParsingFieldUpdateProcessorsTest, AssignTest, TestSolrQueryParserDefaultOperatorResource, SortByFunctionTest, TermVectorComponentDistributedTest, RegexBoostProcessorTest, QueryParsingTest, TestPseudoReturnFields, TestSolrJ, TestReplicationHandler, TestArbitraryIndexDir, ScriptEngineTest, ZkNodePropsTest, TestHashPartitioner, TestJmxIntegration, TestLMJelinekMercerSimilarityFactory, URLClassifyProcessorTest, TestNonDefinedSimilarityFactory, TestDocSet, TestQueryUtils, TestExtendedDismaxParser, TestSchemaVersionResource, TestValueSourceCache, IndexSchemaTest, UpdateParamsTest, AbstractAnalyticsFacetTest, SolrTestCaseJ4Test, OverseerCollectionProcessorTest, DistanceFunctionTest, CustomCollectionTest, HdfsUnloadDistributedZkTest, NoCacheHeaderTest, SOLR749Test, TestAtomicUpdateErrorCases, CollectionsAPIDistributedZkTest, JSONWriterTest, TestSolrQueryParserResource, FileBasedSpellCheckerTest, AnalysisErrorHandlingTest, TestQuerySenderListener, TestHighlightDedupGrouping, RAMDirectoryFactoryTest, CoreAdminCreateDiscoverTest, TestFreeTextSuggestions, CircularListTest, HdfsChaosMonkeySafeLeaderTest, PathHierarchyTokenizerFactoryTest, DocValuesMultiTest, SignatureUpdateProcessorFactoryTest, SuggesterTSTTest, BasicFunctionalityTest, TestJmxMonitoredMap, ConnectionManagerTest, TimeZoneUtilsTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, UnloadDistributedZkTest, OpenCloseCoreStressTest, ShardRoutingTest, BasicZkTest, RecoveryZkTest, TestRandomFaceting, LeaderElectionTest, ZkSolrClientTest, TestZkChroot, TestRandomDVFaceting, ZkCLITest, ShardRoutingCustomTest, SliceStateUpdateTest, AutoCommitTest, TestDistributedGrouping, TestRecovery, ZkControllerTest, TestStressReorder, TestJoin, TestMultiCoreConfBootstrap, TestStressVersions, HardAutoCommitTest, TestCoreContainer, SimpleFacetsTest, SolrCoreTest, StatsComponentTest, SpellCheckComponentTest, TestGroupingSearch, QueryElevationComponentTest, PeerSyncTest, BadIndexSchemaTest, ConvertedLegacyTest, TestSort, DirectUpdateHandlerTest, TestBadConfig, TestIndexSearcher, HighlighterTest, ShowFileRequestHandlerTest, CurrencyFieldOpenExchangeTest, DistributedQueryElevationComponentTest, CurrencyFieldXmlFileTest, TestCoreDiscovery, AnalysisAfterCoreReloadTest, SuggesterFSTTest, CoreAdminHandlerTest, TestFoldingMultitermQuery, DocValuesTest, SuggesterTest, SchemaVersionSpecificBehaviorTest, SolrCoreCheckLockOnStartupTest, TestWordDelimiterFilterFactory, DirectUpdateHandlerOptimizeTest, DefaultValueUpdateProcessorTest, TestRemoteStreaming, TestSolrDeletionPolicy1, DebugComponentTest, DisMaxRequestHandlerTest, StandardRequestHandlerTest, DocumentAnalysisRequestHandlerTest, TestQueryTypes, TestOmitPositions, TermsComponentTest, XmlUpdateRequestHandlerTest, RequestHandlersTest, TestIndexingPerformance, MoreLikeThisHandlerTest, TestSolrQueryParser, LoggingHandlerTest, SolrPluginUtilsTest, TestCollationField, ReturnFieldsTest, UpdateRequestProcessorFactoryTest, MBeansHandlerTest, UniqFieldsUpdateProcessorFactoryTest, BinaryUpdateRequestHandlerTest, PingRequestHandlerTest, CSVRequestHandlerTest, TestComponentsName, CoreContainerCoreInitFailuresTest, TestBinaryResponseWriter, TestLFUCache, HighlighterConfigTest, AlternateDirectoryTest, TestQuerySenderNoQuery, SolrIndexConfigTest, TestStressRecovery, TestSolrDeletionPolicy2, SampleTest, TestBinaryField, TestSearchPerf, NumericFieldsTest, MinimalSchemaTest, TestElisionMultitermQuery, OutputWriterTest, TestFuzzyAnalyzedSuggestions, ExternalFileFieldSortTest, TestSolrCoreProperties, TestPostingsSolrHighlighter, DirectSolrConnectionTest, NotRequiredUniqueKeyTest, TestPhraseSuggestions, TestCharFilters, TestCodecSupport, SynonymTokenizerTest, TestXIncludeConfig, TestDFRSimilarityFactory, TestPerFieldSimilarity, TestLMDirichletSimilarityFactory, TestBM25SimilarityFactory, LegacyHTMLStripCharFilterTest, TestFastOutputStream, TestFastLRUCache, ChaosMonkeyNothingIsSafeTest, PreAnalyzedFieldTest, TestSuggestSpellingConverter, SpellingQueryConverterTest, DOMUtilTest, ClusterStateTest, TestLRUCache, TestUtils, TestDocumentBuilder, SliceStateTest, UUIDFieldTest, FileUtilsTest, TestRTGBase, CursorPagingTest, TestCursorMarkWithoutUniqueKey, AbstractAnalyticsStatsTest, NoFacetTest, ExpressionTest]
   [junit4] Completed in 2.48s, 12 tests, 6 failures <<< FAILURES!

[...truncated 167 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:453: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:433: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:39: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-targets.xml:37: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build.xml:189: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\common-build.xml:491: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:1268: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:901: There were test failures: 372 suites, 1604 tests, 4 errors, 6 failures, 40 ignored (12 assumptions)

Total time: 82 minutes 13 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_60-ea-b04 -XX:-UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure